From ziade.tarek at gmail.com Fri Jan 1 16:07:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:07:20 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04D81D51@TK5EX14MBXC118.redmond.corp.microsoft.com> <94bdd2610912141458m52da21ear4970506a37214e6@mail.gmail.com> <4dab5f760912230700l38a597f7l855bb2304025d6d5@mail.gmail.com> Message-ID: <94bdd2611001010707h4043ff64x7476049e435852b@mail.gmail.com> 2009/12/23 Glyph Lefkowitz : [..] > 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". > > The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. ?In other words it's got "bin", "lib", "share", "etc", etc.. ?~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation. Right, good idea Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Fri Jan 1 16:32:27 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:32:27 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> Message-ID: <94bdd2611001010732o38a563bcu6d0cc9122ed5c88f@mail.gmail.com> On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziad? wrote: > Hello, > > A while ago I've proposed to refactor the APIs that provides access to > the installation paths and configuration variables at runtime into a > single module called "sysconfig", and make it easier for all > implementations to work with them. I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek From status at bugs.python.org Fri Jan 1 18:07:11 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Jan 2010 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100101170711.A5AAF78654@psf.upfronthosting.co.za> ACTIVITY SUMMARY (12/25/09 - 01/01/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2537 open (+22) / 16902 closed (+19) / 19439 total (+41) Open issues with patches: 1016 Average duration of open issues: 705 days. Median duration of open issues: 462 days. Open Issues Breakdown open 2504 (+22) pending 32 ( +0) Issues Created Or Reopened (42) _______________________________ doc: Code is not always colored 12/30/09 CLOSED http://bugs.python.org/issue7487 reopened flox documention buglet for PyBuffer 12/26/09 CLOSED http://bugs.python.org/issue7577 created ronaldoussoren easy Behavior of operations on a closed file object is not documented 12/26/09 http://bugs.python.org/issue7578 created blep Patch to add docstrings to msvcrt 12/26/09 http://bugs.python.org/issue7579 created brian.curtin patch distutils makefile from python.org installer doesn't work on OS 12/27/09 http://bugs.python.org/issue7580 created bskaplan incomplete doc of zlib 12/27/09 CLOSED http://bugs.python.org/issue7581 created coolwanglu [patch] diff.py to use iso timestamp 12/27/09 http://bugs.python.org/issue7582 created techtonik patch doctest should normalize tabs when comparing output 12/27/09 http://bugs.python.org/issue7583 created techtonik datetime.rfcformat() for Date and Time on the Internet 12/27/09 http://bugs.python.org/issue7584 created techtonik [patch] difflib should separate filename from timestamp with tab 12/27/09 http://bugs.python.org/issue7585 created techtonik patch Typo in collections documentation 12/28/09 CLOSED http://bugs.python.org/issue7586 created nneonneo Python 3.1.1 source build issues 12/28/09 CLOSED http://bugs.python.org/issue7587 created sharmaag unittest.TestCase.shortDescription isn't short anymore 12/28/09 http://bugs.python.org/issue7588 created exarkun setup.py shouldn't try to build nis module when nis headers aren 12/28/09 CLOSED http://bugs.python.org/issue7589 created Arfrever patch 'exceptions' module mentioned in documentation 12/28/09 CLOSED http://bugs.python.org/issue7590 created July test_get_platform fails on 3.1 12/28/09 http://bugs.python.org/issue7591 created pitrou buildbot ssl module documentation: SSLSocket.unwrap description shown twi 12/28/09 http://bugs.python.org/issue7592 created mnewman Computed-goto patch for RE engine 12/29/09 http://bugs.python.org/issue7593 created akuchling patch, needs review shlex refactoring 12/29/09 http://bugs.python.org/issue7594 created ferringb patch, needs review Doc typo for select.kevent() 12/29/09 CLOSED http://bugs.python.org/issue7595 created whit537 test_logging sometimes fails 12/29/09 CLOSED http://bugs.python.org/issue7596 created pitrou curses.use_env implementation error 12/30/09 http://bugs.python.org/issue7597 created kanru patch Cannot install, problems with assembly? 12/30/09 CLOSED http://bugs.python.org/issue7598 created sh.00 (c)ElementTree can produce invalid XML 12/30/09 http://bugs.python.org/issue7599 created jwilk interpreter crashes before start 12/30/09 CLOSED http://bugs.python.org/issue7600 created mhh91 Installation - 2 tests failed: test_cmd_line test_xmlrpc 12/30/09 CLOSED http://bugs.python.org/issue7601 created sadhak Doc: make clean and make update do not delete or update Doc/tool 12/30/09 CLOSED http://bugs.python.org/issue7602 created flox There is no 'seq' type 12/30/09 CLOSED http://bugs.python.org/issue7603 created tom66 delattr __slots__ inconsistancy 12/30/09 CLOSED http://bugs.python.org/issue7604 created ferringb test_cmd_line fails with non-ascii path 12/30/09 http://bugs.python.org/issue7605 created pitrou test_xmlrpc fails with non-ascii path 12/30/09 http://bugs.python.org/issue7606 created pitrou stringlib fastsearch could be improved on 64-bit builds 12/30/09 http://bugs.python.org/issue7607 created pitrou PyUnicode_FromFormatV handles %R and %S incorrectly. 12/30/09 CLOSED http://bugs.python.org/issue7608 created alexandre.vassalotti Add --with-system-expat option 12/31/09 CLOSED http://bugs.python.org/issue7609 created Arfrever patch Cannot use both read and readline method in same ZipExtFile obje 12/31/09 http://bugs.python.org/issue7610 created lucifer shlex not posix compliant when parsing "foo#bar" 12/31/09 http://bugs.python.org/issue7611 created jjdmol2 Fix "pass and object" typo in Library Reference / Built-in Types 12/31/09 CLOSED http://bugs.python.org/issue7612 created ygale patch [cppcheck] found a regression : Invalid number of character ((). 12/31/09 CLOSED http://bugs.python.org/issue7613 created ettl.martin patch Python 2.6.4 segfaults 12/31/09 CLOSED http://bugs.python.org/issue7614 created ttsiod unicode_escape codec does not escape quotes 01/01/10 http://bugs.python.org/issue7615 created rhansen patch test_memoryview test_setitem_writable failures with Intel ICC 01/01/10 http://bugs.python.org/issue7616 created ivank distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option 01/01/10 http://bugs.python.org/issue7617 created Arfrever patch Issues Now Closed (46) ______________________ True division of integers could be more accurate 715 days http://bugs.python.org/issue1811 mark.dickinson patch API to clear most free lists 690 days http://bugs.python.org/issue2019 amaury.forgeotdarc patch unit test UnicodeWarning 687 days http://bugs.python.org/issue2100 ezio.melotti test_disutils fails 568 days http://bugs.python.org/issue3054 ronaldoussoren patch test_formatdate_usegmt fails on non-englisch locale 451 days http://bugs.python.org/issue4046 r.david.murray "??" in u"foo" raises a misleading error 410 days http://bugs.python.org/issue4328 r.david.murray zipfile - add __exit__ attribute to make ZipFile object compatib 287 days http://bugs.python.org/issue5511 ezio.melotti patch test_urllib2 fails - urlopen error file not on local host 271 days http://bugs.python.org/issue5625 orsenthil Improve --with-dbmliborder option 170 days http://bugs.python.org/issue6491 benjamin.peterson patch Move the special-case for integer objects out of PyBytes_FromObj 141 days http://bugs.python.org/issue6687 alexandre.vassalotti patch, 26backport setup.py fails to find headers of system libffi 104 days http://bugs.python.org/issue6943 benjamin.peterson patch, easy The replacement suggested for callable(x) in py3k is not equival 92 days http://bugs.python.org/issue7006 benjamin.peterson patch C/API - Document exceptions 89 days http://bugs.python.org/issue7033 lekma patch subprocess.check_output: "docstring has inconsistent leading whi 35 days http://bugs.python.org/issue7381 georg.brandl patch optparse Documentation References Example Files that do not Exis 30 days http://bugs.python.org/issue7404 georg.brandl patch datetime.datetime.isoformat truncation problem 29 days http://bugs.python.org/issue7413 amaury.forgeotdarc patch, needs review doc: Code is not always colored 0 days http://bugs.python.org/issue7487 georg.brandl float('nan')**2 != nan. float('nan')**2 error 33 on windows 13 days http://bugs.python.org/issue7534 mark.dickinson patch If a generator raises TypeError when being unpacked, an unrelate 10 days http://bugs.python.org/issue7548 amaury.forgeotdarc ctypes doc improvement: c_char_p 6 days http://bugs.python.org/issue7569 georg.brandl Strange issue : cursor.commit() with sqlite 5 days http://bugs.python.org/issue7572 ghaering tes_math fails Mac OS X 10.4 due to OverflowError in test_mtestf 5 days http://bugs.python.org/issue7575 mark.dickinson patch documention buglet for PyBuffer 2 days http://bugs.python.org/issue7577 georg.brandl easy incomplete doc of zlib 0 days http://bugs.python.org/issue7581 amaury.forgeotdarc Typo in collections documentation 0 days http://bugs.python.org/issue7586 georg.brandl Python 3.1.1 source build issues 0 days http://bugs.python.org/issue7587 amaury.forgeotdarc setup.py shouldn't try to build nis module when nis headers aren 1 days http://bugs.python.org/issue7589 benjamin.peterson patch 'exceptions' module mentioned in documentation 1 days http://bugs.python.org/issue7590 georg.brandl Doc typo for select.kevent() 0 days http://bugs.python.org/issue7595 georg.brandl test_logging sometimes fails 1 days http://bugs.python.org/issue7596 ezio.melotti Cannot install, problems with assembly? 0 days http://bugs.python.org/issue7598 ezio.melotti interpreter crashes before start 0 days http://bugs.python.org/issue7600 r.david.murray Installation - 2 tests failed: test_cmd_line test_xmlrpc 1 days http://bugs.python.org/issue7601 ezio.melotti Doc: make clean and make update do not delete or update Doc/tool 0 days http://bugs.python.org/issue7602 georg.brandl There is no 'seq' type 0 days http://bugs.python.org/issue7603 benjamin.peterson delattr __slots__ inconsistancy 0 days http://bugs.python.org/issue7604 benjamin.peterson PyUnicode_FromFormatV handles %R and %S incorrectly. 0 days http://bugs.python.org/issue7608 alexandre.vassalotti Add --with-system-expat option 1 days http://bugs.python.org/issue7609 benjamin.peterson patch Fix "pass and object" typo in Library Reference / Built-in Types 0 days http://bugs.python.org/issue7612 ezio.melotti patch [cppcheck] found a regression : Invalid number of character ((). 0 days http://bugs.python.org/issue7613 ezio.melotti patch Python 2.6.4 segfaults 0 days http://bugs.python.org/issue7614 mark.dickinson Fwd: Debian Bug#42318: urllib.py has problems with malformed pro 3436 days http://bugs.python.org/issue210849 shinnosuke urllib / urllib2 should cache 301 redirections 2425 days http://bugs.python.org/issue735515 pitrou fast modular exponentiation 2084 days http://bugs.python.org/issue936813 mark.dickinson patch bdist_deb - Debian packager 316 days http://bugs.python.org/issue1054967 astraw patch Carbon.Scrap.PutScrapFlavor 987 days http://bugs.python.org/issue1700507 ronaldoussoren Top Issues Most Discussed (10) ______________________________ 11 Add Option to Bind to a Local IP Address in httplib.py 462 days open http://bugs.python.org/issue3972 8 fast modular exponentiation 2084 days closed http://bugs.python.org/issue936813 6 [patch] difflib should separate filename from timestamp with ta 5 days open http://bugs.python.org/issue7585 6 [patch] diff.py to use iso timestamp 5 days open http://bugs.python.org/issue7582 6 Implement fastsearch algorithm for rfind/rindex 24 days open http://bugs.python.org/issue7462 5 test_xmlrpc fails with non-ascii path 2 days open http://bugs.python.org/issue7606 5 test_logging sometimes fails 1 days closed http://bugs.python.org/issue7596 5 Patch to add docstrings to msvcrt 6 days open http://bugs.python.org/issue7579 5 Distutils-based installer does not detect 64bit versions of Pyt 126 days open http://bugs.python.org/issue6792 5 _sha256 et al. encode to UTF-8 by default 17 days open http://bugs.python.org/issue3745 From stefan_ml at behnel.de Sun Jan 3 09:11:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jan 2010 09:11:16 +0100 Subject: [Python-Dev] Providing support files to assist 3.x extension authors In-Reply-To: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> References: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> Message-ID: Case Vanhorsen, 20.12.2009 01:38: > When I ported gmpy (Python to GMP multiple precision library) to > Python 3.x, I began to use PyLong_AsLongAndOverflow frequently. I > found the code to slightly faster and cleaner than using PyLong_AsLong > and checking for overflow. You might want to look at the code Cython generates for integer type conversions. We use specialised coercion code that gets generated on-the-fly to convert Python long/int from and to all sorts of C integer types with compile time (portability) and runtime size/value checks. Depending on your needs, this may or may not be faster than the above C-API function. Stefan From david.lyon at preisshare.net Mon Jan 4 00:42:15 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 10:42:15 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> Message-ID: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> > On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: > > This new operator removes the ambiguity the original proposal had, > without making it more > complex for common use cases. So if you dislike it, you will need to > propose something > else that also fixes the ambiguity we had. Ok. > Environment markers >.. >Here are some example of fields using such markers: > >Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: [Windows] pywin32 1.0+ That's simpler, shorter, and less ambiguous. Easier to parse for package managers. David From python at mrabarnett.plus.com Mon Jan 4 01:14:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 04 Jan 2010 00:14:43 +0000 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4132F3.7020905@mrabarnett.plus.com> David Lyon wrote: >> On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: >> >> This new operator removes the ambiguity the original proposal had, >> without making it more >> complex for common use cases. So if you dislike it, you will need to >> propose something >> else that also fixes the ambiguity we had. > > Ok. > >> Environment markers >> .. >> Here are some example of fields using such markers: >> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. > 'win32' is more specific than 'Windows' and, to me, '1.0+' means '>=1.0', not '>1.0'. From martin at v.loewis.de Mon Jan 4 01:21:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jan 2010 01:21:53 +0100 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4134A1.5090203@v.loewis.de> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. Don't you want the PEP to complete? Why this bike-shedding? I can agree it's shorter. I can't agree that it's simpler, or less ambiguous. Regards, Martin From ssteinerx at gmail.com Mon Jan 4 01:29:14 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 3 Jan 2010 19:29:14 -0500 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: On Jan 3, 2010, at 7:21 PM, Martin v. L?wis wrote: >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Really.... Enough, already! S From david.lyon at preisshare.net Mon Jan 4 01:47:56 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 11:47:56 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: <1349.218.214.45.58.1262566076.squirrel@syd-srv02.ezyreg.com> Hi Martin, Happy New Year, >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Well, I'm just helping out by pointing out some simpler ways as Tarek asked me. I was only answering his question. I was out celebrating so it took longer to reply than normal. Bike-Shedding ? Me ? which bikeshed? wanting simple? Anyway, I'm just reading the PEPs and commenting. If there are some alterations that can be done, lets discuss them. > I can agree it's shorter. > .. Cool. What I'd really like is a 'Code-Repository:' keyword so that we can install programs/libraries directly into a system. I feel that this would really simplify the packaging landscape, making it much easier for the scientific community and others to do python software installs. This would allow us to perphaps have something even *more modern* than CPAN. So yes, getting PEP-345 right is important to me. Have a nice day. David From abpillai at gmail.com Mon Jan 4 10:08:05 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 4 Jan 2010 14:38:05 +0530 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: <8548c5f31001040108v635a78ech33521371c6c50e82@mail.gmail.com> On Sun, Dec 27, 2009 at 11:21 PM, anatoly techtonik wrote: > On Tue, Dec 15, 2009 at 12:37 AM, Steven Bethard > wrote: > > > > If you're only concerned about 2.X, then yes, optparse will *never* be > > removed from 2.X. There will be a deprecation note in the 2.X > > documentation but deprecation warnings will only be issued when the -3 > > flag is specified. Please see the "Deprecation of optparse" section of > > the PEP: > > > > http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Ref http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse . Considering that optparse will be deprecated like 5 years from now. I think this point is moot. The deprecation strategy IMHO is perhaps way too conservative. Maybe it should be deprecated faster ;) -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Mon Jan 4 10:32:10 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 4 Jan 2010 10:32:10 +0100 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: >> If you're only concerned about 2.X, then yes, optparse will *never* be >> removed from 2.X. There will be a deprecation note in the 2.X >> documentation but deprecation warnings will only be issued when the -3 >> flag is specified. Please see the "Deprecation of optparse" section of >> the PEP: >> >> http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Compare: > http://docs.python.org/library/optparse.html > http://argparse.googlecode.com/svn/tags/r101/doc/index.html > > argparse should be recommended as advanced and more featured variant > of optparse - that goes without saying, but forcing people to switch > and annoying them with deprecation messages brings only headache. As has been noted already nobody is forcing people to switch. Optparse will be available as a separate package and everybody will be free to install it and will not have any deprecation messages anywhere. > Just > like optparse is better getopt, the latter could also be useful for > people coming from other languages and porting their libraries to > Python. > > I would better concentrate on real code examples how argparse solves > problems and would really want to see > http://www.python.org/dev/peps/pep-0389/#out-of-scope-various-feature-requests > implemented until argparse enters phase where backward incompatible > changes in API are impossible. > > I would prefer to see PEP 389 as a document describing proposed > solutions to argument parsing problems rather than petition to replace > one library with another. So, it should display common argument > parsing scenarios (use cases) with examples that are also useful > recipes for documentation. I guess more than 90% people here doesn't > have time to read argparse methods descriptions to see what they could > be useful for. -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From olemis at gmail.com Mon Jan 4 15:24:05 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 4 Jan 2010 09:24:05 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> Message-ID: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: > Thanks for the quick response. > > On 30/12/2009, Benjamin Peterson wrote: >> >> When I made that change, I didn't know that the __unittest "hack" was >> being used elsewhere outside of unittest, so I felt fine replacing it >> with another. While I still consider it an implementation detail, I >> would be ok with exposing an "official" API for this. Perhaps >> __unittest_ignore_traceback? > > Well, bazaar has had the trick for a couple of years, and googling > around now turns up some other projects using it or thinking about it: > > > > Add `dutest` and probably `nose` [2]_ and ... > Reinstating the old implementation (with the same name) would mean > that existing code would keep working with Python 2.7 IOW ... if it is removed all the libraries will have to change and check for the Py version and ... (probably not a big deal depending on the details of the ?solution?) > but maybe a > discussion could start about a new, less hacky, way of doing the same > I am strongly -1 for modifying the classes in ?traditional? unittest module [2]_ (except that I am strongly +1 for the package structure WITHOUT TOUCHING anything else ...) , and the more I think about it I am more convinced ... but anyway, this not a big deal (and in the end what I think is not that relevant either ... :o). So ... pass > May not be worthwhile making life more complicated though, > there aren't *that* many unittest-extending projects. > But every library extending `unittest` will do it or otherwise not-so-useful error messages will be displayed ;o) PS: Happy New Year ! BTW .. [1] [feature] nosexml should omit trace frames where `__unittest... (http://code.google.com/p/python-nosexml/issues/detail?id=4#c0) .. [2] Assessment of unittest 2.7 API : new features and opinions (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free new ripit 1.3.3 Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/KFwyUTKH0vI/ From juanfhj at gmail.com Tue Jan 5 17:10:16 2010 From: juanfhj at gmail.com (Juan Fernando Herrera J.) Date: Tue, 5 Jan 2010 11:10:16 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility Message-ID: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Jan 5 17:50:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 05 Jan 2010 16:50:15 +0000 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B436DC7.8040605@voidspace.org.uk> On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way > major software hasn't been ported to it. I'm eager to use 3, but > paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me > if this has been suggested in the past. > I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well. I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter). All the best, Michael Foord > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Jan 5 18:21:31 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 5 Jan 2010 11:21:31 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion. I'm sure everyone out there would like to dive into supporting 3.x, but it's tough to prioritize when you are up to your eyeballs with 2.x bugs in your tracker. Look at Twisted ( http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3) -- over 1000 issues, ~5 developers -- 3.x support won't be here tomorrow, but it's on the horizon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 18:23:08 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2010 09:23:08 -0800 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B436DC7.8040605@voidspace.org.uk> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B436DC7.8040605@voidspace.org.uk> Message-ID: On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord wrote: > On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > > I don't know about other developers, but I certainly expected Python 3 > adoption to take a few years due to libraries only gradually making the > jump. I also expected there to be nervousness and pain around the migration > as well. > > I think in fact that libraries *are* migrating and there are lots of > encouraging signs. As a community we need to do all we can to promote Python > 3 adoption, but I don't think there is much reason to be worried (or > complacent for that matter). > Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Jan 5 18:52:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 18:52:20 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <319e029f1001050952o71cb29afh66445550beeb99c2@mail.gmail.com> On Tue, Jan 5, 2010 at 17:10, Juan Fernando Herrera J. wrote: > I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. Yes. Python 3 is not what you want to use today if you write applications. If you write libraries 2010 is the year to port, IMO. With some luck, 2011 will be the year to start porting applications properly. We'll see. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ianb at colorstudy.com Tue Jan 5 20:00:49 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 13:00:49 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin wrote: > On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > >> How about a new python 3 release with (possibly partial) backwards >> compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major >> software hasn't been ported to it. I'm eager to use 3, but paradoxically, >> the 3 release makes me rather stuck with 2.6. Excuse me if this has been >> suggested in the past. >> >> > The proper route to take, in my opinion, is to see what 2.x libraries you > are using that are not 3.x compatible, run 2to3 on them, then run their test > suite, and see where you get. Submit a patch or two to the library and see > what happens -- it at least gets the wheels in motion. > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 5 21:24:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 5 Jan 2010 15:24:45 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> On Jan 5, 2010, at 2:00 PM, Ian Bicking wrote: > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. It seems like this is a problem to be addressed, then. Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. > The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). No-op constructions like 'bytes("")' could help for older versions of Python, though. A very, very small runtime shim could provide support for these, if 2to3 could be told about it somehow. > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? Sort of like our own version of the #line directive? :) Seriously though, I find it hard to believe that this is a big problem. The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. > I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. This is almost certainly more of a pain than trying to trick 2to3 into doing the right thing. From regebro at gmail.com Tue Jan 5 21:57:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 21:57:35 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <319e029f1001051257k3827c52ahcd115b15d9a8ece7@mail.gmail.com> On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz wrote: > It seems like this is a problem to be addressed, then. ?Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x. >> The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated. > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 5 22:07:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 22:07:23 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B43AA0B.5030308@v.loewis.de> > It's not even that easy -- libraries can't apply patches for Python 3 > compatibility as they usually break Python 2 compatibility. Potentially > libraries could apply patches that make a codebase 2to3 ready, but from > what I've seen that's more black magic than straight forward updating, > as such patches have to trick 2to3 producing the output that is desired. I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x. > The only workable workflow I've seen people propose for maintaining a > single codebase with compatibility across both 2 and 3 is to use such > tricks, with aliases to suppress some 2to3 updates when they are > inappropriate I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way. > Also, running 2to3 on installation is kind of annoying, as you get > source that isn't itself the canonical source, so to fix bugs you have > to look at the installed source and trace it back to the bug in the > original source. That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin From martin at v.loewis.de Tue Jan 5 22:13:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jan 2010 22:13:07 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <4B43AB63.3000607@v.loewis.de> > No-op constructions like 'bytes("")' could help for older versions of > Python, though. A very, very small runtime shim could provide > support for these, if 2to3 could be told about it somehow. You actually don't *need* 2to3 support for that - bytes("") can work in either version: 2.x: def bytes(s): return s 3.x: def bytes(s) return s.encode("ascii") On top of that, you *could* as for bytes("") to be converted to b"" in 3.x - and it is indeed possible to tell 2to3 about that, and you don't even need to modify 2to3's source to do so: it can be part of your package. >> Also, running 2to3 on installation is kind of annoying, as you get >> source that isn't itself the canonical source, so to fix bugs you >> have to look at the installed source and trace it back to the bug >> in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in > .pycs rather than based on where the code was actually loaded in the > filesystem, couldn't 2to3 could do .pyc rewriting to point at the > original source? Sort of like our own version of the #line > directive? :) I think it could, but it would be fairly flaky as the pycs can get lost, and lose that information after regeneration. Also, it may be confusing in other scenarios. I'd rather have it generate separate mapper files, and change the traceback printing to consider these (as an option). > Seriously though, I find it hard to believe that this is a big > problem. The 3.x source looks pretty similar to the 2.x source, and > it's good to look at both if you're dealing with a 3.x issue. That's my experience as well - the only challenge is that you can't cut-n-paste the source URL into an editor. Regards, Martin From ianb at colorstudy.com Tue Jan 5 22:39:21 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 15:39:21 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43AA0B.5030308@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: On Tue, Jan 5, 2010 at 3:07 PM, "Martin v. L?wis" wrote: > > > It's not even that easy -- libraries can't apply patches for Python 3 > > compatibility as they usually break Python 2 compatibility. ?Potentially > > libraries could apply patches that make a codebase 2to3 ready, but from > > what I've seen that's more black magic than straight forward updating, > > as such patches have to trick 2to3 producing the output that is desired. > > I wouldn't qualify it in that way. It may be necessary occasionally to > trick 2to3, but that's really a bug in 2to3 which you should report, so > that trickery is then a work-around for a bug - something that you may > have to do with other API, as well. > > The "black magic" is really more in the parts that 2to3 doesn't touch > at all (because they are inherently not syntactic); these are the > problem areas Guido refers to. The "black magic" then is to make the > same code work unmodified for both 2.x and 3.x. Just to clarify, the black magic I'm referring to is things like: try: ?? ?unicode_ = unicode except NameError: ?? ?unicode_ = str and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). ?If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. ?It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. ?For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str. ?That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself. Also replying to Glyph: > > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? ?Sort of like our own version of the #line directive? :) > > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun. So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits. Which is a terribly crotchety of me. Sorry. -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From martin at v.loewis.de Tue Jan 5 23:23:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 23:23:53 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: <4B43BBF9.2000302@v.loewis.de> > Just to clarify, the black magic I'm referring to is things like: > > try: > unicode_ = unicode > except NameError: > unicode_ = str > > and some other aliases like this that are unambiguous and which 2to3 > won't touch (if you write them correctly). If the porting guide noted > all these tricks (of which several have been developed, and I'm only > vaguely aware of a few) that would be helpful. It's not a lot of > tricks, but the tricks are not obvious and 2to3 gets the translation > wrong pretty often without them. For instance, when I say str in > Python 2 I often means bytes, unsurprisingly, but 2to3 translates both > str and unicode to str. No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3? > That *nothing* translates to bytes by default > (AFAIK) means that people must either be living in a bytes-free world > (which sure, lots of code does) or they are using tricks not included > in 2to3 itself. By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3). > Since 2to3 maintains line numbers yes, it wouldn't be that bad. But > then I don't currently develop any code that is "installed", I only > develop code that is directly from a source code checkout, and where > the checkout is put on the path. I guess I could have something that > automatically builds the code on every edit, and that's not > infeasible. It's just not fun. Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it. > So long as I have to support Python 2 > (which is like forever) then adding Python 3 only makes development > that much more complicated and much less fun, with no concrete > benefits. I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin From ziade.tarek at gmail.com Wed Jan 6 01:08:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 01:08:34 +0100 Subject: [Python-Dev] PEP 386 and PEP 345 Message-ID: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Hi, I think we've reached a consensus on those two PEPs. Although, there's one last point that was forgotten in the discussions : I've introduced "rc" in the pre-releases markers, so PEP 386 is compatible with Python's own version scheme. "rc" comes right after "c" in the sorting. It's slightly redundant with the "c" marker but I don't think this really matters as long as consumers know how to order them (a < b < c < rc). I have also stated that "c" is the preferred marker for third party projects, from PEP 386 point of view. Is there anything else I can do to make those two PEPs accepted ? Regards Tarek -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Wed Jan 6 01:26:34 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:26:34 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43BBF9.2000302@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> <4B43BBF9.2000302@v.loewis.de> Message-ID: <1322.218.214.45.58.1262737594.squirrel@syd-srv02.ezyreg.com> Hi Martin, > ... but in the long run, starting now will have saved you a lot of > pressure in the long run, and stop users from switching away from > your packages because of lack of Python 3 support. In a production situation it works the other way around. If there's an application that requires twisted (or whatever package) then most people would use the appropriate interpreter to match the library. Since you guys all did your jobs so well :-) doing so is painless. Because there is so much "comfort" with the existing situation it makes it an effort for people to move to a different lounge chair. Namely python 3. I'd suggest that moving the package set (pypi) to python 3 could be kicked along with the help of some automated tools. I don't know what tools you guys have got. But I am very sure that if code analysis was provided to package developers on python 3 (so they don't have to run it themselves), then it would be like an even bigger tv screen in a bigger lounge room and it would assist in drawing them over. David From david.lyon at preisshare.net Wed Jan 6 01:36:24 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:36:24 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <1337.218.214.45.58.1262738184.squirrel@syd-srv02.ezyreg.com> Hi Ian, > The only workable workflow I've seen people propose for maintaining a > single > codebase with compatibility across both 2 and 3 is to use such tricks, > with > aliases to suppress some 2to3 updates when they are inappropriate, so that > you can run 2to3 on install and have a single canonical Python 2 source. > Python 2.7 won't help much (even though it is trying) as the introduction > of non-ambiguous constructions like b"" aren't compatible with previous > versions of Python and so can't be used in many libraries (support at > least > back to Python 2.5 is the norm for most libraries, I think). > > Also, running 2to3 on installation is kind of annoying, as you get source > that isn't itself the canonical source, so to fix bugs you have to look at > the installed source and trace it back to the bug in the original source. > > I suspect a reasonable workflow might be possible with hg and maybe patch > queues, but I don't feel familiar enough with those tools to map that out. That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David From david.lyon at preisshare.net Wed Jan 6 03:01:22 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 13:01:22 +1100 (EST) Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: <1617.218.214.45.58.1262743282.squirrel@syd-srv02.ezyreg.com> > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? Tarek, Given that I helped out so much last year with the PEP in discussing different options, even if they weren't accepted, I really feel that it is unfair if my name isn't mentioned. It was a huge time sacrifice on my part. For example, even if I only managed to explain the version numbering and clarify how that worked. It did take me some time to do that. What I did do however, was spend a lot of time with the multiplatform "Markers". I still think that two short weeks more of "discussion" could resolve some issues. That discussion went for 4 months on distutils-ml. Look, major issues aside, can you make the following concessions on PEP-345 which I only feel will help it, namely: 1) Source-Repository: specify a code repository to install from 2) Streamline Requires-Python: by implementing "Markers" as noted by the PEP. A marker being something like "Requires-Python(windows): lxml". Otherwise remove the word marker from the PEP and just replace with "metacode". Markers are what were discussed on distutils-ml. Metacode is what is described in the PEP. 3) Remove the inconsistency and platform ambiguities. Especially for windows users. For example, "win32" is extremely confusing for windows users right now. As more and more systems now are 64 bit. Use the platform module, instead of the sys module for constants. I'll post to distutils-ml on this. I am certainly not trying to hold this PEP up, and I apologise on my part for my late attention. I will post to distutils-ml on these and i promise to keep my comments unheated and unwitty. Having said that, PEP-345 is *super-important*. A week or two or three more discussion and the issues can be resolved. We all just want to focus on being productive. It would be a great accomplishment for you to get PEP-345 approved and likewise for me getting mentioned even in a minor role as helping out on a PEP. So I'm hoping that you can make a few last minute concessions meaning that we can all happily go on our way in 2010. Regards David From brett at python.org Wed Jan 6 06:20:30 2010 From: brett at python.org (Brett Cannon) Date: Tue, 5 Jan 2010 21:20:30 -0800 Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 16:08, Tarek Ziad? wrote: > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? > As you said, consensus has been reached, so just Guido's BDFL stamp of approval is all I can think of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Jan 6 12:19:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:19:45 +0000 Subject: [Python-Dev] bug triage Message-ID: <4B4471D1.9020707@simplistix.co.uk> Hi All, Is there a high volume of incoming bugs to the Python tracker? If so, I'd like to help with triaging. I think I have all the necessary access, what I'm missing is the knowledge of how to set myself up to get notifications of new bugs... How do I do that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at voidspace.org.uk Wed Jan 6 12:24:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 06 Jan 2010 11:24:37 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: <4B4472F5.8000709@voidspace.org.uk> On 06/01/2010 11:19, Chris Withers wrote: > Hi All, > > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the > necessary access, what I'm missing is the knowledge of how to set > myself up to get notifications of new bugs... > > How do I do that? > Bug triaging is one of Python's "big needs" and anything you do to help on this score would be much appreciated. Particularly reviewing new and outstanding issues. I assumed there would be RSS feeds for bug tracker activity but can't easily find these on the tracker. There is a bot that posts activity to #python-dev, so there must be some way of getting this information. All the best, Michael > cheers, > > Chris > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Jan 6 12:25:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 11:25:16 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: Hi Chris, > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Do you really want to get such notifications? There may be a lot of them. If you want however, you can join #python-dev on IRC (irc.freenode.net) where there's a bot which posts updates of all bugs on the tracker. There's usually not a lot of discussion going on so you probably won't feel flooded. In addition to bug triage, what is needed is reviewing of existing patches, as well as writing patches for issues which haven't been addressed yet :-) Regards Antoine. From chris at simplistix.co.uk Wed Jan 6 12:30:40 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:30:40 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B447460.7080100@simplistix.co.uk> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. Yeah, email-out is what I'm really after... I have it for my own Roundup instance so it can't be that hard to do ;-) Roch?, you guys host the bug tracker, right? Is there email-out set up for it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Jan 6 12:37:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 21:37:23 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B4475F3.5040406@gmail.com> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. I'm pretty sure the bugs list is still the primary spooled notification mechanism: http://mail.python.org/mailman/listinfo/python-bugs-list There are also the weekly tracker activity summaries that are posted here to python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From chris at simplistix.co.uk Wed Jan 6 12:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:41:28 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <4B4476E8.8050709@simplistix.co.uk> Nick Coghlan wrote: > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list That's what I was after, thanks! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From facundobatista at gmail.com Wed Jan 6 13:03:08 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 6 Jan 2010 09:03:08 -0300 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: On Wed, Jan 6, 2010 at 8:19 AM, Chris Withers wrote: > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Not notifications, but maybe a way to have a higher look of them for easy selection: http://www.taniquetil.com.ar/cgi-bin/pytickets.py Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From ziade.tarek at gmail.com Wed Jan 6 13:29:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:29:58 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord wrote: > On 06/01/2010 11:19, Chris Withers wrote: >> >> Hi All, >> >> Is there a high volume of incoming bugs to the Python tracker? >> If so, I'd like to help with triaging. I think I have all the necessary >> access, what I'm missing is the knowledge of how to set myself up to get >> notifications of new bugs... >> >> How do I do that? >> > > Bug triaging is one of Python's "big needs" and anything you do to help on > this score would be much appreciated. Particularly reviewing new and > outstanding issues. Another useful triage I think, is to review the oldest bugs (some of them are > 5 years) and remove the ones that are not relevant anymore, or duplicate with newer entries. Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Wed Jan 6 13:31:08 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 12:31:08 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B44828C.4070201@simplistix.co.uk> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with someone as a paired task for those 2 days... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Jan 6 13:37:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:37:55 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B44828C.4070201@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B44828C.4070201@simplistix.co.uk> Message-ID: <94bdd2611001060437s2d0242aembc669c3263773fea@mail.gmail.com> On Wed, Jan 6, 2010 at 1:31 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Another useful triage I think, is to review the oldest bugs (some of >> them are > 5 years) >> and remove the ones that are not relevant anymore, or duplicate with >> newer entries. > > I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with > someone as a paired task for those 2 days... I'll be doing Distutils stuff but I can probably help around a bit in that task: Distutils have quite a few old issues so I can tackle those From roche at upfrontsystems.co.za Wed Jan 6 13:32:39 2010 From: roche at upfrontsystems.co.za (=?ISO-8859-1?Q?Roch=E9?= Compaan) Date: Wed, 06 Jan 2010 14:32:39 +0200 Subject: [Python-Dev] bug triage In-Reply-To: <4B447460.7080100@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B447460.7080100@simplistix.co.uk> Message-ID: <1262781159.2836.218.camel@didi> On Wed, 2010-01-06 at 11:30 +0000, Chris Withers wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > Yeah, email-out is what I'm really after... I have it for my own Roundup > instance so it can't be that hard to do ;-) > > Roch?, you guys host the bug tracker, right? Is there email-out set up > for it? We do, but we don't administer it. There are a few administrators taking care of it and you should be able to reach them by logging your request here: http://psf.upfronthosting.co.za/roundup/meta/ or post it to the Infrastructure mailing list: infrastructure at python.org -- Roch? Compaan Upfront Systems http://www.upfrontsystems.co.za From ncoghlan at gmail.com Wed Jan 6 13:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 22:57:24 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B4488B4.2070208@gmail.com> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I believe someone (Daniel Diniz, maybe?) did do a pass over those some time in the last 12 months, so most of the obviously irrelevant ones that are that old should already be gone. Not to say it isn't worth doing another pass, just saying not to get disheartened if there aren't many that can be readily closed. There are at least a few still kicking around just because they're difficult to deal with (there's an ancient one to do with one of the ways circular imports can fail that I occasionally go back and reread before moving on to something more tractable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Wed Jan 6 14:43:17 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 08:43:17 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <4B4476E8.8050709@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> <4B4476E8.8050709@simplistix.co.uk> Message-ID: <20100106134317.3EF141FB5A6@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:41:28 +0000, Chris Withers wrote: > Nick Coghlan wrote: > > I'm pretty sure the bugs list is still the primary spooled notification > > mechanism: > > http://mail.python.org/mailman/listinfo/python-bugs-list > > That's what I was after, thanks! Just for completeness, there's also new-bugs-announce if you want just *new* bug notification. That's more for people who want to watch for bugs they want to become nosy on, though; if you are doing triage python-bugs-list is what you want. Please also read http://www.python.org/dev/workflow/ if you haven't already. Thanks for being willing to chip in! --David From brian.curtin at gmail.com Wed Jan 6 15:57:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 08:57:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4488B4.2070208@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > I believe someone (Daniel Diniz, maybe?) did do a pass over those some > time in the last 12 months, so most of the obviously irrelevant ones > that are that old should already be gone. Not to say it isn't worth > doing another pass, just saying not to get disheartened if there aren't > many that can be readily closed. > > There are at least a few still kicking around just because they're > difficult to deal with (there's an ancient one to do with one of the > ways circular imports can fail that I occasionally go back and reread > before moving on to something more tractable). > > Cheers, > Nick. > On the topic of bugs that can be readily closed (literally), I've recently come across a number of issues which appear to be sitting in a patch or review stage, but their patches have been committed and the issue remains open. What is the best course of action there? I'd just go ahead and close the issue myself but I don't have tracker privileges. I'm willing to help out with another Daniel Diniz-esque triage sweep if that would help. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Wed Jan 6 16:14:20 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 6 Jan 2010 10:14:20 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <7FCF8B19-3465-4392-80CC-BEAEBD926ECA@gmail.com> On Jan 6, 2010, at 7:29 AM, Tarek Ziad? wrote: > On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord > wrote: >> On 06/01/2010 11:19, Chris Withers wrote: >>> >>> Hi All, >>> >>> Is there a high volume of incoming bugs to the Python tracker? >>> If so, I'd like to help with triaging. I think I have all the necessary >>> access, what I'm missing is the knowledge of how to set myself up to get >>> notifications of new bugs... >>> >>> How do I do that? >>> >> >> Bug triaging is one of Python's "big needs" and anything you do to help on >> this score would be much appreciated. Particularly reviewing new and >> outstanding issues. > > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I was actually thinking about that the other day when I saw that the average age of bugs on the Python tracker was at some hideously large 3 digit number. The 'success' statistic would be to bring that down below, say, 100. S From skip at pobox.com Wed Jan 6 17:38:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 6 Jan 2010 10:38:13 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <19268.48245.616046.874126@montanaro.dyndns.org> >>>>> "Nick" == Nick Coghlan writes: Nick> I'm pretty sure the bugs list is still the primary spooled Nick> notification mechanism: Nick> http://mail.python.org/mailman/listinfo/python-bugs-list Actually, there is a new-bugs-announce list: http://mail.python.org/mailman/listinfo/new-bugs-announce Skip From solipsis at pitrou.net Wed Jan 6 18:56:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 17:56:49 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > On the topic of bugs that can be readily closed (literally), I've > recently come across a number of issues which appear to be sitting in a > patch or review stage, but their patches have been committed and the > issue remains open. What is the best course of action there? Post a message on the issue asking for info. From solipsis at pitrou.net Wed Jan 6 19:09:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 18:09:39 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > > On the topic of bugs that can be readily closed (literally), I've > > recently come across a number of issues which appear to be sitting in a > > patch or review stage, but their patches have been committed and the > > issue remains open. What is the best course of action there? > > Post a message on the issue asking for info. Ok, I realize my answer might have been a bit terse :-) The patch might be waiting to be merged in all development branches, or it may not totally resolve the issue, or perhaps documentation needs to be updated, or perhaps it is pending a verdict from the buildbots, etc. You can't deduce that the issue is completely fixed from the simple fact that something has been committed. Regards Antoine Pitrou. From brett at python.org Wed Jan 6 20:03:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 11:03:32 -0800 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > >> I believe someone (Daniel Diniz, maybe?) did do a pass over those some >> time in the last 12 months, so most of the obviously irrelevant ones >> that are that old should already be gone. Not to say it isn't worth >> doing another pass, just saying not to get disheartened if there aren't >> many that can be readily closed. >> >> There are at least a few still kicking around just because they're >> difficult to deal with (there's an ancient one to do with one of the >> ways circular imports can fail that I occasionally go back and reread >> before moving on to something more tractable). >> >> Cheers, >> Nick. >> > > On the topic of bugs that can be readily closed (literally), I've recently > come across a number of issues which appear to be sitting in a patch or > review stage, but their patches have been committed and the issue remains > open. What is the best course of action there? I'd just go ahead and close > the issue myself but I don't have tracker privileges. > > If a core developer is willing to step forward and vouch for you to get tracker privileges then I will give them to you. We are trying to give out tracker privs w/ less time than required to get commit privileges. So as long as you have helped out on a few issues in a positive and correct way that should be enough to get one of the regulars who perform triage to notice. -Brett I'm willing to help out with another Daniel Diniz-esque triage sweep if that > would help. > > Brian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Wed Jan 6 21:41:05 2010 From: nad at acm.org (Ned Deily) Date: Wed, 06 Jan 2010 12:41:05 -0800 Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: In article <4B4475F3.5040406 at gmail.com>, Nick Coghlan wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list Also, that mailing list (along with most python development related mailing lists) is mirrored at gmane.org which means it can also be obtained via a newsreader (NNTP) or various RSS feeds. http://dir.gmane.org/gmane.comp.python.bugs -- Ned Deily, nad at acm.org From nick.bastin at gmail.com Wed Jan 6 22:14:54 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:14:54 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? Message-ID: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> (This may occur on more platforms - I can test on more unix platforms if the consensus is this is an actual problem and I'm not just a nut) On freebsd5, if you do a simple ./configure --enable-shared in current (2.7) trunk, your python shared library will build properly, but all modules will fail to find the shared library and thus fail to build: gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.7/u1/Python/Python-2.7a1/Modules/_struct.o -L/u1/tmp/python2.7a1/lib -L/usr/local/lib -lpython2.7 -o build/lib.freebsd-5.3-RELEASE-i386-2.7/_struct.so /usr/bin/ld: cannot find -lpython2.7 building '_ctypes_test' extension ... This of course is because libpython2.7.so is in the current directory and not (yet) installed in /usr/local/lib. I've made a very simple fix for this problem that works, but at least to me smells a bit funny, which is to modify setup.py to add the following to detect_modules(): # If we did --enable-shared, we need to be able to find the library # we just built in order to build the modules. if platform == 'freebsd5': add_dir_to_list(self.compiler_obj.library_dirs, '.') Which brings me to a few questions: a) Does this seem like a real problem, or am I missing something obvious? b) Does this fix seem like the sensible thing to do? (it seems at least that we ought to check that the user configured --enable-shared and only set -L. in that case, if that's possible) Setting --enable-shared when you actually have a libpython2.7.so in /usr/local/lib (or whatever --prefix you've selected) is possibly even more dangerous, because it may succeed in linking against a differently-built library than what you intended. -- Nick From nick.bastin at gmail.com Wed Jan 6 22:39:17 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:39:17 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <66d0a6e11001061339t38202b70mca1091e1926ecf4b@mail.gmail.com> On Wed, Jan 6, 2010 at 16:14, Nicholas Bastin wrote: > This of course is because libpython2.7.so is in the current directory > and not (yet) installed in /usr/local/lib. One minor correction - as you could see from the compile line, the actual --prefix in this case is /u1/tmp/python2.7a1, but the libraries obviously aren't installed there yet either. Perhaps a better fix than setting -L. would be to put the shared library in build/lib.freebsd-5.3-RELEASE-i386-2.7 and add that to the library path for the linker (the build creates this directory, but installs nothing in it). -- Nick From martin at v.loewis.de Wed Jan 6 23:21:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jan 2010 23:21:44 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <4B450CF8.8090009@v.loewis.de> > b) Does this fix seem like the sensible thing to do? No. Linking in setup.py should use the same options as if the module was built as *shared* through Modules/Setup, which, IIUC, should use BLDLIBRARY. Regards, Martin From nick.bastin at gmail.com Thu Jan 7 01:08:13 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 19:08:13 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B450CF8.8090009@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> Message-ID: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> On Wed, Jan 6, 2010 at 17:21, "Martin v. L?wis" wrote: >> b) Does this fix seem like the sensible thing to do? > > No. Linking in setup.py should use the same options as if the module > was built as *shared* through Modules/Setup, which, IIUC, should use > BLDLIBRARY. Thanks for that pointer, that makes much more sense. Indeed, BLDLIBRARY on FreeBSD* is set to '-L. -lpython$(VERSION)' if you set --enable-shared, but somehow that piece of information doesn't propagate into the module build. More investigation to be done... -- Nick From rdmurray at bitdance.com Thu Jan 7 02:22:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 20:22:42 -0500 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > > On the topic of bugs that can be readily closed (literally), I've recently > > come across a number of issues which appear to be sitting in a patch or > > review stage, but their patches have been committed and the issue remains > > open. What is the best course of action there? I'd just go ahead and close > > the issue myself but I don't have tracker privileges. > > > > > If a core developer is willing to step forward and vouch for you to get > tracker privileges then I will give them to you. We are trying to give out > tracker privs w/ less time than required to get commit privileges. So as > long as you have helped out on a few issues in a positive and correct way > that should be enough to get one of the regulars who perform triage to > notice. > > -Brett I've done a quick scan of issues Brian is nosy on to refresh my memory, and I'd say he's definitely been making positive contributions. I'm willing to volunteer to keep an eye on his triage work for a while if you grant him tracker privs. Brian, I assume you'll be cognizant of Antoine's advice about making sure a bug really should be closed before closing it :) Hanging out in #python-dev on freenode while working on issues can be helpful, as well, since you can quickly ask whoever is there for second opinions on particular bugs. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Thu Jan 7 02:28:26 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 17:28:26 -0800 Subject: [Python-Dev] bug triage In-Reply-To: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > > On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > > On Wed, Jan 6, 2010 at 06:57, Brian Curtin > wrote: > > > On the topic of bugs that can be readily closed (literally), I've > recently > > > come across a number of issues which appear to be sitting in a patch or > > > review stage, but their patches have been committed and the issue > remains > > > open. What is the best course of action there? I'd just go ahead and > close > > > the issue myself but I don't have tracker privileges. > > > > > > > > If a core developer is willing to step forward and vouch for you to get > > tracker privileges then I will give them to you. We are trying to give > out > > tracker privs w/ less time than required to get commit privileges. So as > > long as you have helped out on a few issues in a positive and correct way > > that should be enough to get one of the regulars who perform triage to > > notice. > > > > -Brett > > I've done a quick scan of issues Brian is nosy on to refresh my > memory, and I'd say he's definitely been making positive contributions. > I'm willing to volunteer to keep an eye on his triage work for a while > if you grant him tracker privs. > > Done for the username brian.curtin (email doesn't match the one Brian emailed from so do let me know, Brian if this is the right username). Welcome aboard! -Brett > Brian, I assume you'll be cognizant of Antoine's advice about making > sure a bug really should be closed before closing it :) Hanging out in > #python-dev on freenode while working on issues can be helpful, as well, > since you can quickly ask whoever is there for second opinions on > particular bugs. > > -- > R. David Murray www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Jan 7 02:59:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 19:59:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 19:28, Brett Cannon wrote: > > > On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > >> >> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: >> > On Wed, Jan 6, 2010 at 06:57, Brian Curtin >> wrote: >> > > On the topic of bugs that can be readily closed (literally), I've >> recently >> > > come across a number of issues which appear to be sitting in a patch >> or >> > > review stage, but their patches have been committed and the issue >> remains >> > > open. What is the best course of action there? I'd just go ahead and >> close >> > > the issue myself but I don't have tracker privileges. >> > > >> > > >> > If a core developer is willing to step forward and vouch for you to get >> > tracker privileges then I will give them to you. We are trying to give >> out >> > tracker privs w/ less time than required to get commit privileges. So as >> > long as you have helped out on a few issues in a positive and correct >> way >> > that should be enough to get one of the regulars who perform triage to >> > notice. >> > >> > -Brett >> >> I've done a quick scan of issues Brian is nosy on to refresh my >> memory, and I'd say he's definitely been making positive contributions. >> I'm willing to volunteer to keep an eye on his triage work for a while >> if you grant him tracker privs. >> >> > Done for the username brian.curtin (email doesn't match the one Brian > emailed from so do let me know, Brian if this is the right username). > Welcome aboard! > > Yep, that's the one. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Jan 7 04:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 07 Jan 2010 03:07:56 +0000 Subject: [Python-Dev] GIL required for _all_ Python calls? Message-ID: <4B45500C.8090905@mrabarnett.plus.com> Hi, I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? Or is it just a case of checking the source files for mentions of the GIL? The header file for PyList_New, for example, doesn't mention it! Thanks From john.arbash.meinel at gmail.com Thu Jan 7 04:25:48 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 06 Jan 2010 21:25:48 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B45543C.2090200@gmail.com> MRAB wrote: > Hi, > > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? Or is it just a case of checking the source > files for mentions of the GIL? The header file for PyList_New, for > example, doesn't mention it! > > Thanks Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may get concurrent updating of the value, and then the final value is wrong. (two threads do 5+1 getting 6, rather than 7, and when the decref, you end up at 4 rather than back at 5). AFAIK, the only things that don't require the GIL are macro functions, like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for example, will be increfing and setting the exception state, so certainly needs the GIL to be held. John =:-> From benjamin at python.org Thu Jan 7 04:32:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jan 2010 21:32:17 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45543C.2090200@gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> Message-ID: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> 2010/1/6 John Arbash Meinel : > Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may > get concurrent updating of the value, and then the final value is wrong. > (two threads do 5+1 getting 6, rather than 7, and when the decref, you > end up at 4 rather than back at 5). Correct. > > AFAIK, the only things that don't require the GIL are macro functions, > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > example, will be increfing and setting the exception state, so certainly > needs the GIL to be held. As a general rule, I would say, no Py* macros are safe without the gil either (the exception being Py_END_ALLOW_THREADS), since they mutate Python objects which must be protected. -- Regards, Benjamin From guido at python.org Thu Jan 7 05:29:03 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Jan 2010 20:29:03 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: On Wed, Jan 6, 2010 at 7:32 PM, Benjamin Peterson wrote: > 2010/1/6 John Arbash Meinel : > > AFAIK, the only things that don't require the GIL are macro functions, > > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > > example, will be increfing and setting the exception state, so certainly > > needs the GIL to be held. > > As a general rule, I would say, no Py* macros are safe without the gil > either (the exception being Py_END_ALLOW_THREADS), since they mutate > Python objects which must be protected. That's keeping it on the safe side, since there are some macros like PyString_AS_STRING() that are also safe, *if* you are owning at least one reference to the string object. At the same time, "no Py* macros" is not quite strong enough, since if you called PyString_AS_STRING() before releasing the GIL but you don't own a reference to the string object, the string might be deallocated behind your back by another thread. A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). -- --Guido van Rossum (python.org/~guido) From yoann.padioleau at facebook.com Thu Jan 7 08:24:42 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Wed, 6 Jan 2010 23:24:42 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt Message-ID: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a file called ast.txt. Where does this file come from ? Is it generated from Python.asdl ? From johan.gill at agama.tv Thu Jan 7 10:46:52 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 10:46:52 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. Message-ID: <4B45AD8C.5000405@agama.tv> Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before, so this is probably an already solved problem. Regards Johan Gill Agama Technologies From regebro at gmail.com Thu Jan 7 12:12:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 12:12:17 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45AD8C.5000405@agama.tv> References: <4B45AD8C.5000405@agama.tv> Message-ID: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I imagine that other companies have contributed before, so this is > probably an already solved problem. I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hodgestar+pythondev at gmail.com Thu Jan 7 12:28:01 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 7 Jan 2010 13:28:01 +0200 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon From swamy.sangamesh at gmail.com Thu Jan 7 11:46:59 2010 From: swamy.sangamesh at gmail.com (swamy sangamesh) Date: Thu, 7 Jan 2010 16:16:59 +0530 Subject: [Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9 Message-ID: Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have attached the full test case result. If i change the type c_int and c_short to c_unit and c_ushort of class "BITS(Structure)" in file test_bitfields.py then no failures are seen. Has anyone faced the similar issue or any help is appreciated. Thanks, Sangamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ctype-testcases Type: application/octet-stream Size: 22996 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 7 13:23:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 07 Jan 2010 22:23:11 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: <4B45D22F.40404@gmail.com> Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls we need to think >> about? I imagine that other companies have contributed before, so this is >> probably an already solved problem. > > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license without requesting additional permission from the copyright holder. For corporate contributions, I believe the contributor agreement needs to be signed by an authorised agent of the company - the place to check that would probably be psf at python.org (that's the email address for the PSF board). Assuming the subject line relates to the code that you would like to contribute though, that particular change is unlikely to happen - 2.6 is in maintenance mode and changing RLock from a Python implementation to the faster C one is solidly in new feature territory. Although a backport of the 3.2 C RLock implementation to 2.7 could be useful, I doubt that backporting code provided by an existing committer would be the subject of this query :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Thu Jan 7 14:11:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:11:27 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. Even if the contributed code as in this case is a method in an existing file? How does that work, how do they keep ownership of one method in the threading.py method? :-) > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) Ah. I probably misunderstood what the suggested contribution was. Maybe it was a separate file, which I didn't get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Thu Jan 7 14:15:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 07 Jan 2010 13:15:09 +0000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> Message-ID: <4B45DE5D.3030104@voidspace.org.uk> On 07/01/2010 13:11, Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > >> As Simon pointed out, while some organisations do work that way, the PSF >> isn't one of them. >> >> The PSF only requires that the code be contributed under a license that >> then allows us to turn around and redistribute it under a different open >> source license without requesting additional permission from the >> copyright holder. >> > Even if the contributed code as in this case is a method in an > existing file? How does that work, how do they keep ownership of one > method in the threading.py method? :-) > > When contributing code to Python all work remains copyright the original author. The combined work is copyright *everyone*. The PSF has a license to distribute it, which is all that is important. How you meaningfully exercise your ownership over chunks of code is left for the reader to determine... (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From stefan_ml at behnel.de Thu Jan 7 14:30:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:30:16 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: Guido van Rossum, 07.01.2010 05:29: > A better rule would be "you may access the memory buffer in a PyString > or PyUnicode object with the GIL released as long as you own a > reference to the string object." Everything else is out of bounds (or > not worth the bother). Is that a "yes" regarding the OP's original question about releasing the GIL during regexp searches? Stefan From regebro at gmail.com Thu Jan 7 14:36:42 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:36:42 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DE5D.3030104@voidspace.org.uk> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> <4B45DE5D.3030104@voidspace.org.uk> Message-ID: <319e029f1001070536t49f76899j111212b02c14f192@mail.gmail.com> On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Thu Jan 7 15:05:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:05:23 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB, 07.01.2010 04:07: > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. > or PyErr_SetString? Certainly not safe. > Is there an easy way to find out? Release it and fix any crashes? Note that this isn't a safe solution, though, as some GIL requiring code may be platform specific. So a better approach might be to extract any obviously problematic stuff from the existing code (such as any exception handling, explicit ref-counting or object creation), and *then* try to release the GIL. Stefan From solipsis at pitrou.net Thu Jan 7 16:38:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 15:38:31 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or macros you want to call with the GIL released, and assess whether it is safe to call them. As already pointed out, no reference count should be changed, and generally no mutable container should be accessed, except if that container is known not to be referenced anywhere else (that would be the case for e.g. a list that your function has created and is busy populating). I agree that releasing the GIL when doing non-trivial regex searches is a worthwhile research, so please don't give up immediately :-) Regards Antoine Pitrou. From olemis at gmail.com Thu Jan 7 19:24:59 2010 From: olemis at gmail.com (Olemis Lang) Date: Thu, 7 Jan 2010 13:24:59 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> Message-ID: <24ea26601001071024m2abd988fuc77f94845d57483a@mail.gmail.com> On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am strongly -1 for modifying the classes in ?traditional? unittest > module [2]_ (except that I am strongly +1 for the package structure > WITHOUT TOUCHING anything else ...) , and the more I think about it I > am more convinced ... but anyway, this not a big deal (and in the end > what I think is not that relevant either ... :o). So ... > IOW, if I had all the freedom to implement it, after the pkg structure I'd do something like : {{{ #!python class TestResult: # Everything just the same def _is_relevant_tb_level(self, tb): return '__unittest' in tb.tb_frame.f_globals class BetterTestResult(TestResult): # Further code ... maybe ;o) # def _is_relevant_tb_level(self, tb): # This or anything else you might want to do ;o) # globs = tb.tb_frame.f_globals is_relevant = '__name__' in globs and \ globs["__name__"].startswith("unittest") del globs return is_relevant }}} that's what inheritance is for ;o) ... but quite probably that's not gonna happen, just a comment . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Ubuntu sustituye GIMP por F-Spot - http://feedproxy.google.com/~r/simelo-es/~3/-g48D6T6Ojs/ubuntu-sustituye-gimp-por-f-spot.html From martin at v.loewis.de Thu Jan 7 21:24:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:24:32 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: <4B464300.2020204@v.loewis.de> >> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question about releasing the > GIL during regexp searches? No, because the regex engine may also operate on buffers that start moving around when you release the GIL. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:27:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:27:09 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B46439D.9030604@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in place - if their contents changes, regex results may be confusing. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:31:21 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Message-ID: <4B464499.4020305@v.loewis.de> > I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process a file > called ast.txt. Where does this file come from ? Is it generated from > Python.asdl ? astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. HTH, Martin From foom at fuhm.net Thu Jan 7 21:36:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 7 Jan 2010 15:36:37 -0500 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B46439D.9030604@v.loewis.de> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> On Jan 7, 2010, at 3:27 PM, Martin v. L?wis wrote: >> I've been wondering whether it's possible to release the GIL in the >> regex engine during matching. > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Even if they stay in place - > if their contents changes, regex results may be confusing. It seems probably worthwhile to optimize for the common case of using the regexp engine on an immutable object of type "str" or "bytes", and allow releasing the GIL in *that* case, even if you have to keep it for the general case. James From martin at v.loewis.de Thu Jan 7 21:45:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:45:42 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> Message-ID: <4B4647F6.2060309@v.loewis.de> >>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even if they stay in place - >> if their contents changes, regex results may be confusing. > > It seems probably worthwhile to optimize for the common case of using > the regexp engine on an immutable object of type "str" or "bytes", and > allow releasing the GIL in *that* case, even if you have to keep it for > the general case. Right. This problem was the one that I thought of first. Thinking about these things is fairly difficult (to me, at least), so I think I could only tell whether I would consider a patch thread-safe that released the GIL around matching under selected circumstances - if I had the patch available. I don't see any obvious reason (assuming Guido's list of conditions holds - i.e. you are holding references to everything you access). Regards, Martin From ncoghlan at gmail.com Thu Jan 7 21:48:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Jan 2010 06:48:05 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DECB.7070907@agama.tv> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <4B45DECB.7070907@agama.tv> Message-ID: <4B464885.40806@gmail.com> Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a contributor agreement was essential for this particular contribution would depend on how much new code was needed for the backport, but the bulk of the copyright on the C RLock code would remain with Antoine regardless. However, sorting through the legalities of the contributor agreement really is the best way to make sure every is squared away nice and neatly from a legal point of view. After all, even if I was a lawyer (which I'm not, I'm just a developer with an interest in licensing issues), I still wouldn't be *your* lawyer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Jan 7 21:43:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 20:43:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the representation isn't moved away until the buffer is released. Regards Antoine. From martin at v.loewis.de Thu Jan 7 21:59:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:59:29 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B464B31.7040406@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin From johan.gill at agama.tv Thu Jan 7 14:16:59 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 14:16:59 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <4B45DECB.7070907@agama.tv> On 01/07/2010 01:23 PM, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. For corporate contributions, I believe the contributor > agreement needs to be signed by an authorised agent of the company - the > place to check that would probably be psf at python.org (that's the email > address for the PSF board). > > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) > > Regards, > Nick. > > Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Regards Johan Gill Agama Technologies From yoann.padioleau at facebook.com Thu Jan 7 22:07:55 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Thu, 7 Jan 2010 13:07:55 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464499.4020305@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> Message-ID: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> On Jan 7, 2010, at 12:31 PM, Martin v. L?wis wrote: >> I would like to use astgen.py to generate python classes corresponding to the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take as an argument a .asdl file, but apparently it instead process a file >> called ast.txt. Where does this file come from ? Is it generated from >> Python.asdl ? > > astgen.py is not used to process asdl files; ast.txt lives right next to > astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a reflection mechanism things that were generated from ast.txt, then there could be a mismatch. Where does ast.txt comes from ? Shouldn't it be generated itself from Python.adsl ? So we would have Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py containing all the UnarySub, Expression, classes that represents a Python AST. > > HTH, > Martin From martin at v.loewis.de Thu Jan 7 22:11:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 07 Jan 2010 22:11:36 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <4B464E08.5020703@v.loewis.de> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the representation isn't moved away until the buffer is > released. In 2.7, we currently get the buffer with bf_getreadbuffer. In 3.x, we have /* Release the buffer immediately --- possibly dangerous but doing something else would require some re-factoring */ PyBuffer_Release(&view); Even if we do use the new API, and correctly, it still might be confusing if the contents of the buffer changes underneath. Regards, Martin From martin at v.loewis.de Thu Jan 7 22:16:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 22:16:12 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> Message-ID: <4B464F1C.7020404@v.loewis.de> >> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose as a reflection mechanism things that were generated from > ast.txt, then there could be a mismatch. Where does ast.txt comes > from ? Shouldn't it be generated itself from Python.adsl ? What you may not be aware of is that Tools/compiler (and the compiler package that it builds on) are both unused and unmaintained. If the package stops working correctly - tough luck. > So we would have > > Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py > containing all the UnarySub, Expression, classes that represents a > Python AST. No - what actually happens in Python 3.x is this: both the compiler package and Tools/compiler are removed. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 01:10:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 01:10:35 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM Message-ID: <201001080110.35800.victor.stinner@haypocalc.com> Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading an UTF-8 text file including a BOM: #7185 (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with the UTF-8-SIG encoding, but it's possible to do better. I propose to improve open() (TextIOWrapper) by using the BOM to choose the right encoding. I think that only files opened in read only mode should support this new feature. *Read* the BOM in a *write* only file would cause unexpected behaviours. Since my proposition changes the result TextIOWrapper.read()/readline() for files starting with a BOM, we might introduce an option to open() to enable the new behaviour. But is it really needed to keep the backward compatibility? I wrote a proof of concept attached to the issue #7651. My patch only changes the behaviour of TextIOWrapper for reading files starting with a BOM. It doesn't work yet if a seek() is used before the first read. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Jan 8 01:52:20 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 16:52:20 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner wrote: > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". > > See recent issues related to reading an UTF-8 text file including a BOM: #7185 > (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with > the UTF-8-SIG encoding, but it's possible to do better. > > I propose to improve open() (TextIOWrapper) by using the BOM to choose the > right encoding. I think that only files opened in read only mode should > support this new feature. *Read* the BOM in a *write* only file would cause > unexpected behaviours. > > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? > > I wrote a proof of concept attached to the issue #7651. My patch only changes > the behaviour of TextIOWrapper for reading files starting with a BOM. It > doesn't work yet if a seek() is used before the first read. > > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Fri Jan 8 03:23:08 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 02:23:08 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46970C.9010306@mrabarnett.plus.com> Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > Alternatively, have a universal UTF-8/16/32 encoding, ie one that expects UTF-8, with or without BOM, or UTF-16/32 with BOM. > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> >> See recent issues related to reading an UTF-8 text file including a BOM: #7185 >> (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with >> the UTF-8-SIG encoding, but it's possible to do better. >> >> I propose to improve open() (TextIOWrapper) by using the BOM to choose the >> right encoding. I think that only files opened in read only mode should >> support this new feature. *Read* the BOM in a *write* only file would cause >> unexpected behaviours. >> >> Since my proposition changes the result TextIOWrapper.read()/readline() for >> files starting with a BOM, we might introduce an option to open() to enable >> the new behaviour. But is it really needed to keep the backward compatibility? >> >> I wrote a proof of concept attached to the issue #7651. My patch only changes >> the behaviour of TextIOWrapper for reading files starting with a BOM. It >> doesn't work yet if a seek() is used before the first read. >> From nick.bastin at gmail.com Fri Jan 8 04:11:03 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 7 Jan 2010 22:11:03 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> Message-ID: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs to do this on FreeBSD as well (adding bsd to the list of platforms for which this is performed "solves" the problem, but I don't pretend to know enough about either distutils or freebsd to determine if this is the correct solution). -- Nick From glyph at twistedmatrix.com Fri Jan 8 04:34:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 7 Jan 2010 22:34:36 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? It *is* crazy, but unfortunately rather common. Wikipedia has a good description of the issues: . Basically, some Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as being UTF-8, so it's become a convention to do that. That's not good enough, so you need to guess the encoding as well to make sure, but if there is a BOM and you can otherwise verify that the file is probably UTF-8 encoded, you should discard it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Jan 8 05:17:28 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 07 Jan 2010 23:17:28 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solaris and automatically adds '.' to the library_dirs, and > I suspect it just needs to do this on FreeBSD as well (adding bsd to > the list of platforms for which this is performed "solves" the > problem, but I don't pretend to know enough about either distutils or > freebsd to determine if this is the correct solution). I wouldn't say it needed discussion on the SIG: just create a bug report, with the tentative patch you have worked out, and get it assigned to Tarek. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGsdQACgkQ+gerLs4ltQ5BMQCgtV8snMXH/6dDwgdN4sIJljLd koYAoKq6c0tKsRSrITHcygu4Od9FVzF5 =BJaE -----END PGP SIGNATURE----- From guido at python.org Fri Jan 8 05:21:04 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 20:21:04 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > > file starting with a BOM, read()/readline() returns also the BOM whereas the > > BOM should be "ignored". > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > > It *is* crazy, but unfortunately rather common. ?Wikipedia has a good > description of the issues: > . ?Basically, some > Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as > being UTF-8, so it's become a convention to do that. ?That's not good > enough, so you need to guess the encoding as well to make sure, but if there > is a BOM and you can otherwise verify that the file is probably UTF-8 > encoded, you should discard it. That doesn't make sense. If the file isn't UTF-8 you can't see the BOM, because the BOM itself is UTF-8-encoded. (And yes, I know this happens. Doesn't mean we need to auto-guess by default; there are lots of issues e.g. what should happen after seeking to offset 0?) -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Fri Jan 8 07:06:16 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 08 Jan 2010 15:06:16 +0900 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from the user for each instance. But it should do something sane when reading such files. I can't really see any harm in throwing it away, especially since use of ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated IIRC. From tseaver at palladion.com Fri Jan 8 07:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 01:12:12 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> >> BOM should be "ignored". >> >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. > > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) The BOM should not be seekeable if the file is opened with the proposed "guess encoding from BOM" mode: it isn't properly part of the stream at all in that case. A UTF-8 BOM is an absurditiy, but it exists *everywhere* in the wild: Python would do wll to make it as easy as possible to consume such files, as well as the non-insane versions (UTF-16 / UTF-32 BOMs). In the best of all possible worlds, I would just try opening the file so: f = open('/path/to/file', 'r', encoding="DWIFM") and any BOM present would set the encoding for the remainder of the stream.. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGzLsACgkQ+gerLs4ltQ5+cwCdGfycPdj6+cPfD23vH644SpHL sI0AoLGD7nfgMEJdJhBr90yjQQHfDgcJ =js+2 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Fri Jan 8 08:55:27 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 8 Jan 2010 02:55:27 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. It's just that the UTF-8 decoding of the BOM at the start of a file should be "". > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I think it's pretty clear that the BOM should still be skipped in that case ... From martin at v.loewis.de Fri Jan 8 10:05:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:05:17 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F54D.9090103@v.loewis.de> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I think what Glyph meant is this: if a file starts with the UTF-8 signature, assume it's UTF-8. Then validate the assumption against the rest of the file also, and then process it as UTF-8. If the rest clearly is not UTF-8, assume that the UTF-8 signature is bogus. I understood this proposal as a general processing guideline, not something the io library should do (but, say, a text editor). FWIW, I'm personally in favor of using the UTF-8 signature. If people consider them crazy talk, that may be because UTF-8 can't possibly have a byte order - hence I call it a signature, not the BOM. As a signature, I don't consider it crazy at all. There is a long tradition of having magic bytes in files (executable files, Postscript, PDF, ... - see /etc/magic). Having a magic byte sequence for plain text to denote the encoding is useful and helps reducing moji-bake. This is the reason it's used on Windows: notepad would normally assume that text is in the ANSI code page, and for compatibility, it can't stop doing that. So the UTF-8 signature gives them an exit strategy. Regards, Martin From martin at v.loewis.de Fri Jan 8 10:06:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:06:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <201001080110.35800.victor.stinner@haypocalc.com> <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4B46F59A.5060905@v.loewis.de> > But it should do something sane when reading such files. I can't > really see any harm in throwing it away, especially since use of > ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated > IIRC. And indeed it does, when you open the file in the utf-8-sig encoding. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 10:08:30 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 10:08:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46970C.9010306@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46970C.9010306@mrabarnett.plus.com> Message-ID: <201001081008.30904.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 03:23:08, MRAB a ?crit : > Guido van Rossum wrote: > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > > talk. And for the other two, perhaps it would make more sense to have > > a separate encoding-guessing function that takes a binary stream and > > returns a text stream wrapping it with the proper encoding? > > Alternatively, have a universal UTF-8/16/32 encoding, ie one that > expects UTF-8, > with or without BOM, or UTF-16/32 with BOM. Do you mean open(filename, encoding="BOM")? I suppose that "BOM" would be a magical value specific to read a text file (open(filename, "r")), not a real codec? Otherwise which encoding should be used for open(filename, "w", encoding="BOM")? -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Fri Jan 8 10:10:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:10:23 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F67F.60604@v.loewis.de> > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". It depends. If you use the utf-8-sig encoding, it *will* ignore the UTF-8 signature. > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? Absolutely. And there is no need to produce a new option, but instead use the existing options: define an encoding that auto-detects the encoding from the family of BOMs. Maybe you call it encoding="sniff". Regards, Martin From martin at v.loewis.de Fri Jan 8 10:11:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:11:51 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: <4B46F6D7.1050301@v.loewis.de> Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. I'm fairly skeptical that anybody on distutils SIG is interested in details of the Python build process... Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 11:27:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:27:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081127.44044.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : (...) > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I wrote a new version of my patch (version 3): * don't change the default behaviour: use open(filename, encoding="BOM") to check the BOM is there is any * fix for seek(0): always ignore the BOM * add an unit test: check that the right encoding is detect, but also the the BOM is ignored (especially after a seek(0)) BOM encoding doesn't work for writing into a file, so open(filename, "w", encoding="BOM") raises a ValueError. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:31:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:31:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081131.37492.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 01:52:20, Guido van Rossum a ?crit : > And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? I choosed to modify open()+TextIOWrapper instead of writing a new function because I would like to avoid an extra read operation (syscall) on the file. With my implementation, no specific read operation is needed to detect the BOM. The BOM is simply checked in the first _read_chunk(). -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:40:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:40:28 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Improve_open=28=29_to_support_reading?= =?iso-8859-1?q?_file_starting_with=09an_unicode_BOM?= In-Reply-To: <4B46F67F.60604@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> Message-ID: <201001081140.28124.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : > > Builtin open() function is unable to open an UTF-16/32 file starting with > > a BOM if the encoding is not specified (raise an unicode error). For an > > UTF-8 file starting with a BOM, read()/readline() returns also the BOM > > whereas the BOM should be "ignored". > > It depends. If you use the utf-8-sig encoding, it *will* ignore the > UTF-8 signature. Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to remove the BOM after the first read (much harder if you use a module like ConfigParser or csv). > > Since my proposition changes the result TextIOWrapper.read()/readline() > > for files starting with a BOM, we might introduce an option to open() to > > enable the new behaviour. But is it really needed to keep the backward > > compatibility? > > Absolutely. And there is no need to produce a new option, but instead > use the existing options: define an encoding that auto-detects the > encoding from the family of BOMs. Maybe you call it encoding="sniff". Good idea, I choosed open(filename, encoding="BOM"). -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri Jan 8 15:27:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:27:42 +0000 (UTC) Subject: [Python-Dev] GIL required for _all_ Python calls? References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: > > Even if we do use the new API, and correctly, it still might be > confusing if the contents of the buffer changes underneath. Well, no more confusing than when you compute a SHA1 hash or zlib- compress the buffer, is it? Regards Antoine From solipsis at pitrou.net Fri Jan 8 15:34:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:34:26 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any Well, I think if we implement this the default behaviour *should* be changed. It looks a bit senseless to have two different "auto-choose" options, one with encoding=None and one with encoding="BOM". Regards Antoine. From guido at python.org Fri Jan 8 16:48:44 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:48:44 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 6:27 AM, Antoine Pitrou wrote: > Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: >> >> Even if we do use the new API, and correctly, it still might be >> confusing if the contents of the buffer changes underneath. > > Well, no more confusing than when you compute a SHA1 hash or zlib- > compress the buffer, is it? That depends. Algorithms that make exactly one pass over the buffer will run fine (maybe producing a meaningless result). But the regex matcher may scan the buffer repeatedly (for backtracking purposes) and it would take a considerable analysis to prove that cannot mess up its internal data structures if the data underneath changes. (I give it a decent chance that it's fine, but since it was written without ever considering this possibility I'm not 100% sure.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:52:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:52:48 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 11:55 PM, Glyph Lefkowitz wrote: > I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. And I'm saying that it is, with as much certainty as we can ever guess the encoding of a file. > If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. ?It's just that the UTF-8 decoding of the BOM at the start of a file should be "". Sure, a Latin-1-encoded file could start with the same pattern that is a UTF-8-encoded BOM. But at that point, a UTF-16-encoded file is also valid Latin-1. The question was in the context of encoding-guessing; if we're guessing, a UTF-8-encoded BOM cannot signify anything else but UTF-8. (Ditto for UTF-16 and UTF-32 BOMs.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:54:15 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:54:15 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> >> I wrote a new version of my patch (version 3): >> >> ?* don't change the default behaviour: use open(filename, encoding="BOM") to >> check the BOM is there is any > > Well, I think if we implement this the default behaviour *should* be changed. > It looks a bit senseless to have two different "auto-choose" options, one with > encoding=None and one with encoding="BOM". Well there *are* two different auto options: use the environment variables (LANG etc.) or inspect the contents of the file. I think it would be useful to have ways to specify both. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:56:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:56:46 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 1:05 AM, "Martin v. L?wis" wrote: >>> It *is* crazy, but unfortunately rather common. ?Wikipedia has a good >>> description of the issues: >>> . ?Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. ?That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. > > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Sure. I said "crazy talk" only to stir up discussion. Which worked. :-) Also, I don't want Python's default behavior to change -- sniffing the encoding should be a separate option. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:59:45 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:59:45 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: > The BOM should not be seekeable if the file is opened with the proposed > "guess encoding from BOM" mode: ?it isn't properly part of the stream at > all in that case. This feels about right to me. There are still questions though: immediately after opening a file with a BOM, what should .tell() return? And regardless of that, .seek(0) should put the file in that same initial state. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Jan 8 17:03:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:03:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > > Well, I think if we implement this the default behaviour *should* be changed. > > It looks a bit senseless to have two different "auto-choose" options, one with > > encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Yes, perhaps. In the context of open() however I think it would be helpful to change the default. Moreover, reading the BOM is certainly much more reliable than our current guessing based on the locale or the "device encoding". Regards Antoine. From mal at egenix.com Fri Jan 8 17:25:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 17:25:22 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B475C72.1010207@egenix.com> Guido van Rossum wrote: > On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: >> Victor Stinner haypocalc.com> writes: >>> >>> I wrote a new version of my patch (version 3): >>> >>> * don't change the default behaviour: use open(filename, encoding="BOM") to >>> check the BOM is there is any >> >> Well, I think if we implement this the default behaviour *should* be changed. >> It looks a bit senseless to have two different "auto-choose" options, one with >> encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Shouldn't this encoding guessing be a separate function that you call on either a file or a seekable stream ? After all, detecting encodings is just as useful to have for non-file streams. You'd then avoid having to stuff everything into a single function call and also open up the door for more complex application specific guess work or defaults. The whole process would then have two steps: 1. guess encoding import codecs encoding = codecs.guess_file_encoding(filename) 2. open the file with the found encoding f = open(filename, encoding=encoding) For seekable streams f, you'd have: 1. guess encoding import codecs encoding = codecs.guess_stream_encoding(f) 2. wrap the stream with a reader for the found encoding reader_class = codecs.getreader(encoding) g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Jan 8 17:31:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:31:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver palladion.com> wrote: > > The BOM should not be seekeable if the file is opened with the proposed > > "guess encoding from BOM" mode: it isn't properly part of the stream at > > all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? tell() in the context of text I/O is specified to return an "opaque cookie". So whatever value it returns would probably be fine, as long as seeking to that value leaves the file in an acceptable state. Rewinding (seeking to 0) in the presence of a BOM is already reasonably supported by the TextIOWrapper object: >>> dec = codecs.getincrementaldecoder('utf-16')() >>> dec.decode(b'\xff\xfea\x00b\x00') 'ab' >>> dec.decode(b'\xff\xfea\x00b\x00') '\ufeffab' >>> >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' There are tests for this in test_io.py (test_encoded_writes, line 1929, and test_append_bom and test_seek_bom, line 2045). Regards Antoine. From python at mrabarnett.plus.com Fri Jan 8 17:47:18 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 16:47:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081127.44044.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B476196.4080409@mrabarnett.plus.com> Victor Stinner wrote: > Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : > (...) >> (And yes, I know this happens. Doesn't mean we need to auto-guess by >> default; there are lots of issues e.g. what should happen after >> seeking to offset 0?) > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any > * fix for seek(0): always ignore the BOM > * add an unit test: check that the right encoding is detect, but also the the > BOM is ignored (especially after a seek(0)) > > BOM encoding doesn't work for writing into a file, so open(filename, "w", > encoding="BOM") raises a ValueError. > I think it's similar to universal newline mode. You can tell it that you're reading UTF-something-encoded text (common forms only). The preference is UTF-8, but it could be UTF-8-sig (from Windows), or possibly UTF-16/32, which really need a BOM because there are multiple bytes per codepoint, so the bytes could be big-endian or little-endian. The BOM (or signature) tells you what the encoding is, defaulting to UTF-8 if there's none. If it subsequently raises a DecodeError, then so be it! Maybe there should also be a way of determining what encoding it decided it was, so that you can then write a new file in that same encoding. From status at bugs.python.org Fri Jan 8 18:07:13 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Jan 2010 18:07:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100108170714.014B078669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/01/10 - 01/08/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2544 open (+27) / 16937 closed (+15) / 19481 total (+42) Open issues with patches: 1017 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2509 (+27) pending 34 ( +0) Issues Created Or Reopened (43) _______________________________ Extended slicing with classic class behaves strangely 01/07/10 http://bugs.python.org/issue7532 reopened mark.dickinson patch optparse library documentation has an insignificant formatting i 01/01/10 CLOSED http://bugs.python.org/issue7618 created vazovsky patch imaplib shouldn't use cause DeprecationWarnings in 2.6 01/01/10 CLOSED http://bugs.python.org/issue7619 created djc Vim syntax highlight 01/02/10 http://bugs.python.org/issue7620 created july patch Test issue 01/02/10 CLOSED http://bugs.python.org/issue7621 created georg.brandl [patch] improve unicode methods: split() rsplit() and replace() 01/03/10 http://bugs.python.org/issue7622 created flox patch PropertyType missing in Lib/types.py 01/03/10 CLOSED http://bugs.python.org/issue7623 created wplappert isinstance(... ,collections.Callable) fails with oldstyle class 01/03/10 http://bugs.python.org/issue7624 created rgammans bytearray needs more tests for "b.some_method()[0] is not b" 01/03/10 http://bugs.python.org/issue7625 created flox patch Entity references without semicolon in HTMLParser 01/03/10 CLOSED http://bugs.python.org/issue7626 created stefan.schweizer mailbox.MH.remove() lock handling is broken 01/04/10 http://bugs.python.org/issue7627 created sraustein round() doesn't work correctly in 3.1.1 01/04/10 CLOSED http://bugs.python.org/issue7628 created bkovt Compiling with mingw32 gcc, content of variable "extra_postargs" 01/04/10 CLOSED http://bugs.python.org/issue7629 created popelkopp Strange behaviour of decimal.Decimal 01/04/10 CLOSED http://bugs.python.org/issue7630 created parmax undefined label: bltin-file-objects 01/04/10 CLOSED http://bugs.python.org/issue7631 created ezio.melotti dtoa.c: oversize b in quorem 01/04/10 http://bugs.python.org/issue7632 created skrah decimal.py: type conversion in context methods 01/04/10 http://bugs.python.org/issue7633 created skrah patch, easy next/previous links in documentation skip some sections 01/05/10 CLOSED http://bugs.python.org/issue7634 created gagenellina 19.6 xml.dom.pulldom doc: stub? 01/05/10 http://bugs.python.org/issue7635 created tjreedy Add a set update action to optparse 01/05/10 CLOSED http://bugs.python.org/issue7636 created hardkrash patch Improve 19.5. xml.dom.minidom doc 01/05/10 http://bugs.python.org/issue7637 created tjreedy Counterintuitive str.splitlines() inconsistency. 01/05/10 CLOSED http://bugs.python.org/issue7638 created vencabot_teppoo bdist_msi fails on files with long names 01/05/10 http://bugs.python.org/issue7639 created mmm77 buffered io seek() buggy 01/05/10 http://bugs.python.org/issue7640 created pakal patch Built-in Formatter accepts undocumented presentation type 01/06/10 CLOSED http://bugs.python.org/issue7641 created lrekucki [patch] Minor improvement in os.system doc 01/06/10 http://bugs.python.org/issue7642 created Val patch What is an ASCII linebreak? 01/06/10 http://bugs.python.org/issue7643 created flox bug in nntplib.body() method with possible fix 01/06/10 http://bugs.python.org/issue7644 created mdmullins easy test_distutils fails on Windows XP 01/06/10 http://bugs.python.org/issue7645 created austin987 test_telnetlib fails in Windows XP 01/06/10 http://bugs.python.org/issue7646 created austin987 Add statvfs flags to the posix module 01/06/10 http://bugs.python.org/issue7647 created ajax at redhat.com patch test_urllib2 fails on Windows if not run from C: 01/06/10 http://bugs.python.org/issue7648 created austin987 easy "u'%c' % char" broken for chars in range '\x80'-'\xFF' 01/07/10 http://bugs.python.org/issue7649 created ezio.melotti patch, easy test_uuid is invalid 01/07/10 http://bugs.python.org/issue7650 created austin987 Python3: guess text file charset using the BOM 01/07/10 http://bugs.python.org/issue7651 created haypo patch Merge C version of decimal into py3k. 01/07/10 http://bugs.python.org/issue7652 created mark.dickinson Fix description of the way the PythonPath Windows registry key w 01/07/10 CLOSED http://bugs.python.org/issue7653 created BoarGules patch, needs review [patch] Enable additional bytes and memoryview tests. 01/07/10 http://bugs.python.org/issue7654 created flox patch PEP 374 Title usage & script redirection typo fixes 01/07/10 CLOSED http://bugs.python.org/issue7655 created bab9e9 patch test_hashlib fails on some installations (specifically Neal's re 01/08/10 http://bugs.python.org/issue7656 created r.david.murray test_ctypes failure on AIX 5.3 01/08/10 http://bugs.python.org/issue7657 created mallayya patch OS X pythonw.c compile error with 10.4 or earlier deployment tar 01/08/10 http://bugs.python.org/issue7658 created ned.deily Problems with attribute assignment on object instances 01/08/10 http://bugs.python.org/issue7659 created pakal Issues Now Closed (40) ______________________ shutil fails when copying to NTFS in Linux 762 days http://bugs.python.org/issue1545 benjamin.peterson patch Test 751 days http://bugs.python.org/issue1619 georg.brandl Windows Registry issue with 2.5.2 AMD64 msi 645 days http://bugs.python.org/issue2539 loewis Extension module build fails for MinGW: missing vcvarsall.bat 618 days http://bugs.python.org/issue2698 Daniel26 optparse print_usage(),.. methods are not documented 542 days http://bugs.python.org/issue3340 ezio.melotti reference leaks in test_distutils 500 days http://bugs.python.org/issue3660 ezio.melotti patch _sha256 et al. encode to UTF-8 by default 20 days http://bugs.python.org/issue3745 lemburg 26backport Add Option to Bind to a Local IP Address in httplib.py 464 days http://bugs.python.org/issue3972 gregory.p.smith patch, easy no reference for optparse methods 388 days http://bugs.python.org/issue4635 ezio.melotti PyArg_Parse* should raise TypeError for float parsed with intege 339 days http://bugs.python.org/issue5080 mark.dickinson patch Don't use PyLong_SHIFT with _PyLong_AsScaledDouble() 282 days http://bugs.python.org/issue5576 mark.dickinson patch PyFrame_GetLineNumber 244 days http://bugs.python.org/issue5954 georg.brandl patch PyCode_NewEmpty 244 days http://bugs.python.org/issue5959 georg.brandl patch Add non-command help topics to help completion of cmd.Cmd 241 days http://bugs.python.org/issue5991 georg.brandl patch make error 231 days http://bugs.python.org/issue6067 georg.brandl no longer possible to hash arrays 229 days http://bugs.python.org/issue6071 benjamin.peterson patch zipfile: Invalid argument when opening zero-sized files 173 days http://bugs.python.org/issue6511 r.david.murray use different mechanism for pythonw on osx 127 days http://bugs.python.org/issue6834 ned.deily easy Py3k doc: "from __future__ import division" not necessary 32 days http://bugs.python.org/issue7432 ezio.melotti cPickle: stack underflow in load_pop() 31 days http://bugs.python.org/issue7455 pitrou patch crash in str.rfind() with an invalid start value 25 days http://bugs.python.org/issue7458 pitrou patch Implement fastsearch algorithm for rfind/rindex 26 days http://bugs.python.org/issue7462 ezio.melotti patch GZipFile.readline too slow 20 days http://bugs.python.org/issue7471 pitrou patch ssl module documentation: SSLSocket.unwrap description shown twi 5 days http://bugs.python.org/issue7592 georg.brandl Installation - 2 tests failed: test_cmd_line test_xmlrpc 7 days http://bugs.python.org/issue7601 ezio.melotti optparse library documentation has an insignificant formatting i 2 days http://bugs.python.org/issue7618 ezio.melotti patch imaplib shouldn't use cause DeprecationWarnings in 2.6 1 days http://bugs.python.org/issue7619 djc Test issue 0 days http://bugs.python.org/issue7621 ezio.melotti PropertyType missing in Lib/types.py 0 days http://bugs.python.org/issue7623 benjamin.peterson Entity references without semicolon in HTMLParser 2 days http://bugs.python.org/issue7626 r.david.murray round() doesn't work correctly in 3.1.1 0 days http://bugs.python.org/issue7628 benjamin.peterson Compiling with mingw32 gcc, content of variable "extra_postargs" 0 days http://bugs.python.org/issue7629 r.david.murray Strange behaviour of decimal.Decimal 0 days http://bugs.python.org/issue7630 mark.dickinson undefined label: bltin-file-objects 0 days http://bugs.python.org/issue7631 pitrou next/previous links in documentation skip some sections 0 days http://bugs.python.org/issue7634 georg.brandl Add a set update action to optparse 3 days http://bugs.python.org/issue7636 r.david.murray patch Counterintuitive str.splitlines() inconsistency. 0 days http://bugs.python.org/issue7638 flox Built-in Formatter accepts undocumented presentation type 0 days http://bugs.python.org/issue7641 eric.smith Fix description of the way the PythonPath Windows registry key w 0 days http://bugs.python.org/issue7653 r.david.murray patch, needs review PEP 374 Title usage & script redirection typo fixes 0 days http://bugs.python.org/issue7655 georg.brandl patch Top Issues Most Discussed (10) ______________________________ 22 [patch] improve unicode methods: split() rsplit() and replace() 5 days open http://bugs.python.org/issue7622 14 Test suite emits many DeprecationWarnings when -3 is enabled 91 days open http://bugs.python.org/issue7092 13 unicode_escape codec does not escape quotes 8 days open http://bugs.python.org/issue7615 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 0 days open http://bugs.python.org/issue7658 8 Merge C version of decimal into py3k. 1 days open http://bugs.python.org/issue7652 8 "u'%c' % char" broken for chars in range '\x80'-'\xFF' 2 days open http://bugs.python.org/issue7649 8 decimal.py: type conversion in context methods 4 days open http://bugs.python.org/issue7633 8 Patch for [ 735515 ] urllib2 should cache 301 redir 906 days open http://bugs.python.org/issue1755841 7 Test issue 0 days closed http://bugs.python.org/issue7621 7 Cannot use both read and readline method in same ZipExtFile obj 8 days open http://bugs.python.org/issue7610 From tseaver at palladion.com Fri Jan 8 22:09:54 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:09:54 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: >> The BOM should not be seekeable if the file is opened with the proposed >> "guess encoding from BOM" mode: it isn't properly part of the stream at >> all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? And regardless of that, .seek(0) should put the file in that > same initial state. I think the behavior should be something like: >>> f = open('/path/to/maybe-BOM-encoded-file', 'r', encoding='BOM') >>> f.tell() 0L >>> f.seek(-1) >>> f.tell() # count of unicode chars in decoded stream 45L >>> f.seek(0) >>> f.read(1) # read first unicode char decoded from stream. 'A' In other words, the BOM is not readable / seekable at all: it is invisible to the consumer of the decoded stream. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHnyIACgkQ+gerLs4ltQ6s3QCgznD+7FbUzfCbe5TS6OcoXjMg rdgAoJAMEXe2xwLCIwJaZ6XA6rVyTIAi =oXb3 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:19:10 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:19:10 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B475C72.1010207@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: > Shouldn't this encoding guessing be a separate function that you call > on either a file or a seekable stream ? > > After all, detecting encodings is just as useful to have for non-file > streams. Other stream sources typically have out-of-band ways to signal the encoding: only when reading from the filesystem do we pretty much *have* to guess, and in that case the BOM / signature is the best heuristic we have. Also, some non-file streams are not seekable, and so can't be guessed via a pre-pass. > You'd then avoid having to stuff everything into > a single function call and also open up the door for more complex > application specific guess work or defaults. > > The whole process would then have two steps: > > 1. guess encoding > > import codecs > encoding = codecs.guess_file_encoding(filename) Filename is not enough information: or do you mean that API to actually open the stream? > 2. open the file with the found encoding > > f = open(filename, encoding=encoding) > > For seekable streams f, you'd have: > > 1. guess encoding > > import codecs > encoding = codecs.guess_stream_encoding(f) > > 2. wrap the stream with a reader for the found encoding > > reader_class = codecs.getreader(encoding) > g = reader_class(f) > Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoU4ACgkQ+gerLs4ltQ5o3QCeLOJ7J91E+5f66vhgu1BUhYh4 9UgAnR2IeCd0BCsPez8ZilGNHJfhRn3Y =SoPb -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:14:59 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:14:59 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> It *is* crazy, but unfortunately rather common. Wikipedia has a good >>> description of the issues: >>> . Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. If the programmer opens the file using a "guess using the BOM" encoding, Python should *not* attempt to verify that the file is properly encoded: it should check for (and consume) any BOM, and then return a stream which uses the encoding inferred from the BOM. Any errors should be handled later, when characters are read, exactly as if the file had been opened with the same encoding guessed from the BOM. > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Agreed. Having that marker at the start of the file makes interop with other tools *much* easier. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoFMACgkQ+gerLs4ltQ73dACffwUfyh6Q9vUnKYf367QFjNcU RRMAoNuKCWEx7j+MSdTv+UjhAPynBc14 =uAX6 -----END PGP SIGNATURE----- From eric at trueblade.com Fri Jan 8 22:40:47 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 8 Jan 2010 16:40:47 -0500 (EST) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? >> >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. But what if the file were in (for example) a zip file? I think you definitely want to have access to this functionality outside of open(). Eric. From foom at fuhm.net Fri Jan 8 22:49:23 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 8 Jan 2010 16:49:23 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Jan 8, 2010, at 4:14 PM, Tres Seaver wrote: >> I understood this proposal as a general processing guideline, not >> something the io library should do (but, say, a text editor). >> >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly >> have >> a byte order - hence I call it a signature, not the BOM. As a >> signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote >> the >> encoding is useful and helps reducing moji-bake. This is the reason >> it's >> used on Windows: notepad would normally assume that text is in the >> ANSI >> code page, and for compatibility, it can't stop doing that. So the >> UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop > with > other tools *much* easier. Putting the BOM at the beginning of UTF-8 text files is not a good idea, it makes interop much *worse* on a unix system, not better. Without the BOM, most commands do the right thing with UTF-8 text. E.g. to concatenate two files: $ cat file-1 file-2 > file-3 With a BOM at the beginning of the file, it won't work right. Of course, you could modify "cat" (and every other stream processing command) to know how to consume and emit BOMs, and omit the extra one that would show up in the middle of the stream...but even that can't work; what about: $ (cat file-1; cat file-2) > file-3. Should the shell now know that when you run multiple commands, it should eat the BOM emitted from the second command? Basically, using a BOM in a utf-8 file is just not a good idea: it completely ruins interop with every standard unix tool. This is not to say that Python shouldn't have a way to read a file with a UTF-8 BOM: it just shouldn't encourage you to *write* such files. James From mal at egenix.com Fri Jan 8 22:51:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 22:51:26 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <4B47A8DE.1080401@egenix.com> Tres Seaver wrote: > M.-A. Lemburg wrote: > >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? > >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. Sure there are non-seekable file streams, but at least when using StringIO-type streams you don't have that restriction. An encoding detection function would provide more use in other cases as well, so instead of hiding away the functionality in the open() constructor, I'm suggesting to make expose it via the codecs module. Applications can then use it where necessary and also provide their own defaults, using other heuristics. >> You'd then avoid having to stuff everything into >> a single function call and also open up the door for more complex >> application specific guess work or defaults. > >> The whole process would then have two steps: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_file_encoding(filename) > > Filename is not enough information: or do you mean that API to actually > open the stream? Yes. The API would open the file, guess the encoding and then close it again. If you don't want that to happen, you could use the second API I mentioned below on the already open file. Note that this function could detect a lot more encodings with reasonably high probability than just BOM-prefixed ones, e.g. we could also add support to detect encoding declarations such as the ones we use in Python source files. >> 2. open the file with the found encoding > >> f = open(filename, encoding=encoding) > >> For seekable streams f, you'd have: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_stream_encoding(f) I forgot to mention: This API needs to position the file pointer to the start of the first data byte. >> 2. wrap the stream with a reader for the found encoding > >> reader_class = codecs.getreader(encoding) >> g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Fri Jan 8 22:59:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:59:04 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote: >>> Shouldn't this encoding guessing be a separate function that you call >>> on either a file or a seekable stream ? >>> >>> After all, detecting encodings is just as useful to have for non-file >>> streams. >> Other stream sources typically have out-of-band ways to signal the >> encoding: only when reading from the filesystem do we pretty much >> *have* to guess, and in that case the BOM / signature is the best >> heuristic we have. Also, some non-file streams are not seekable, and so >> can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). If the application expects a possibly-BOM-signature-marked file, but you pass it mismatched garbage: >>> f = open('some.zip', encoding='BOM") the error handling should be the same as if you passed any other mismatched encoding: >>> f = open('some.zip', encoding='UTF8') i.e., you discover the error when you try to read from the (non)encoded stream, not when you open it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHqpwACgkQ+gerLs4ltQ7uAACeKEc+WT4TASGcVl1Hfqe6L9La I6EAn1pJtngtLWPdothGbYB+zUabEvTW =TjBK -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Fri Jan 8 23:10:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 23:10:32 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: <201001082310.33029.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 22:40:47, Eric Smith a ?crit : > >> Shouldn't this encoding guessing be a separate function that you call > >> on either a file or a seekable stream ? > >> > >> After all, detecting encodings is just as useful to have for non-file > >> streams. > > > > Other stream sources typically have out-of-band ways to signal the > > encoding: only when reading from the filesystem do we pretty much > > *have* to guess, and in that case the BOM / signature is the best > > heuristic we have. Also, some non-file streams are not seekable, and so > > can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). FYI my patch (encoding="BOM") is implemented in TextIOWrapper, and TextIOWrapper takes a binary stream as input, not a filename. -- Victor Stinner http://www.haypocalc.com/ From yoann.padioleau at facebook.com Fri Jan 8 23:59:52 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Fri, 8 Jan 2010 14:59:52 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464F1C.7020404@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: On Jan 7, 2010, at 1:16 PM, Martin v. L?wis wrote: >>> astgen.py is not used to process asdl files; ast.txt lives right >>> next to astgen.py. Instead, the asdl file is processed by >>> Parser/asdl_c.py. >> >> Yes, I know that. That's why I asked about the relation between >> ast.txt and Python.adsl. If internally the parser uses the .adsl, but >> expose as a reflection mechanism things that were generated from >> ast.txt, then there could be a mismatch. Where does ast.txt comes >> from ? Shouldn't it be generated itself from Python.adsl ? > > What you may not be aware of is that Tools/compiler (and the > compiler package that it builds on) are both unused and unmaintained. I see. So if people want to analyze python code they have to use other tools (like rope?) ? or use reflection ? > > If the package stops working correctly - tough luck. > >> So we would have >> >> Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py >> containing all the UnarySub, Expression, classes that represents a >> Python AST. > > No - what actually happens in Python 3.x is this: both the compiler > package and Tools/compiler are removed. Ok. I will then create my own ast classes generator. Thanks. > > Regards, > Martin From g.brandl at gmx.net Sat Jan 9 00:10:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 09 Jan 2010 00:10:24 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: Am 08.01.2010 22:14, schrieb Tres Seaver: >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly have >> a byte order - hence I call it a signature, not the BOM. As a signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote the >> encoding is useful and helps reducing moji-bake. This is the reason it's >> used on Windows: notepad would normally assume that text is in the ANSI >> code page, and for compatibility, it can't stop doing that. So the UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop with > other tools *much* easier. Except if only 50% of the other tools support the signature. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jan 9 00:57:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 00:57:46 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: <4B47C67A.3020302@v.loewis.de> > I see. So if people want to analyze python code they have to use > other tools (like rope?) ? or use reflection ? Correct. One such tool might be the true Python compiler, along with the _ast module. Regards, Martin From victor.stinner at haypocalc.com Sat Jan 9 00:59:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 00:59:00 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM Message-ID: <201001090059.00848.victor.stinner@haypocalc.com> Hi, Thanks for all the answers! I will try to sum up all ideas here. (1) Change default open() behaviour or make it optional? Guido would like to add an option and keep open() unchanged. He wrote that checking for BOM and using system locale are too much different to be the same option (encoding=None). Antoine would like to check BOM by default, because both options (system locale vs checking for BOM) is the same thing. About Antoine choice (encoding=None): which file modes would check for a BOM? I would like to answer only the read only mode, but then open(filename, "r") and open(filename, "r+") would behave differently? => 1 point for Guido (encoding="BOM" is more explicit) Antoine choice has the advantage of directly support UTF-8+BOM, UTF-16 and UTF-32 for all applications and all modules using open(filename). => 1 point for Antoine (2) Check for a BOM while reading or detect it before? Everybody agree that checking BOM is an interesting option and should not be limited to open(). Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file name or a binary file-like object: it returns the encoding and seek to the file start or just after the BOM. I dislike this function because it requires extra file operations (open (optional), read() and seek()) and it doesn't work if the file is not seekable (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to avoid extra file operations. Note: I implemented the BOM check in TextIOWrapper; so it's already usable for any file-like object. (3) tell() and seek() on a text file starting with a BOM To be consistent with Antoine example: >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' TextIOWrapper: * tell() should return zero at file start, * seek(0) should go be to file start, * and the BOM should always be "ignored". I mean: with open("utf8bom.txt", encoding="BOM") as fp: assert fp.tell() == 0 text = fp.read() # no BOM here fp.seek(0) assert fp.read() == text -- About my patch: - BOM check is explicit: open(filebame, encoding="BOM") - tell() / seek(0) works as expected - BOM bytes are always skipped in read() / readlines() result Hum, I don't know if this email can be called a sum up ;-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 01:09:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 00:09:48 +0000 (UTC) Subject: [Python-Dev] Quick sum up about open() + BOM References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: Hello Victor, Victor Stinner haypocalc.com> writes: > > (1) Change default open() behaviour or make it optional? > [...] > > Antoine would like to check BOM by default, because both options (system > locale vs checking for BOM) is the same thing. To be clear, I am not saying it is the same thing. What I think is that it would be a mistake to use a mildly unreliable heuristic by default (the locale + device encoding heuristic) but refuse to trust a more reliable heuristic (the BOM-based detection algorithm). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Jan 9 01:14:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 00:14:39 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47CA6F.5000607@voidspace.org.uk> On 09/01/2010 00:09, Antoine Pitrou wrote: > Hello Victor, > > Victor Stinner haypocalc.com> writes: > >> (1) Change default open() behaviour or make it optional? >> >> > [...] > >> Antoine would like to check BOM by default, because both options (system >> locale vs checking for BOM) is the same thing. >> > To be clear, I am not saying it is the same thing. What I think is that it would > be a mistake to use a mildly unreliable heuristic by default (the locale + > device encoding heuristic) but refuse to trust a more reliable heuristic (the > BOM-based detection algorithm). > I concur. On Windows both UTF-8 and signature are very common, yet the platform default is the truly awful CP1252. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From floris.bruynooghe at gmail.com Sat Jan 9 01:26:05 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 9 Jan 2010 00:26:05 +0000 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B46F6D7.1050301@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> <4B46F6D7.1050301@v.loewis.de> Message-ID: <20100109002605.GB13536@laurie.devork> On Fri, Jan 08, 2010 at 10:11:51AM +0100, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > I think this problem probably needs to move over to distutils-sig, as > > it doesn't seem to be specific to the way that Python itself uses > > distutils. > > I'm fairly skeptical that anybody on distutils SIG is interested in > details of the Python build process... Uh, hum. Unfounded skepticism. ;-) But as said filing a bug sounds better in this case. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From v+python at g.nevcal.com Sat Jan 9 01:47:38 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 16:47:38 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47D22A.8070305@g.nevcal.com> On approximately 1/8/2010 3:59 PM, came the following characters from the keyboard of Victor Stinner: > Hi, > > Thanks for all the answers! I will try to sum up all ideas here. One concern I have with this implementation encoding="BOM" is that if there is no BOM it assumes UTF-8. That is probably a good assumption in some circumstances, but not in others. * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE encoded files include a BOM. It is only required that UTF-16 and UTF-32 (cases where the endianness is unspecified) contain a BOM. Hence, it might be that someone would expect a UTF-16LE (or any of the formats that don't require a BOM, rather than UTF-8), but be willing to accept any BOM-discriminated format. * Potentially, this could be expanded beyond the various Unicode encodings... one could envision that a program whose data files historically were in any particular national language locale, could want to be enhance to accept Unicode, and could declare that they will accept any BOM-discriminated format, but want to default, in the absence of a BOM, to the original national language locale that they historically accepted. That would provide a migration path for their old data files. So the point is, that it might be nice to have "BOM-otherEncodingForDefault" for each other encoding that Python supports. Not sure that is the right API, but I think it is expressive enough to handle the cases above. Whether the cases solve actual problems or not, I couldn't say, but they seem like reasonable cases. It would, of course, be nicest if OS metadata had been invented way back when, for all OSes, such that all text files were flagged with their encoding... then languages could just read the encoding and do the right thing! But we live in the real world, instead. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From python at mrabarnett.plus.com Sat Jan 9 02:12:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 09 Jan 2010 01:12:28 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B47D7FC.4070703@mrabarnett.plus.com> Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. > > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. > What about listing the possible encodings? It would try each in turn until it found one where the BOM matched or had no BOM: my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') or is that taking it too far? From martin at v.loewis.de Sat Jan 9 02:23:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 02:23:07 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47CA6F.5000607@voidspace.org.uk> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> Message-ID: <4B47DA7B.6050505@v.loewis.de> >>> Antoine would like to check BOM by default, because both options >>> (system locale vs checking for BOM) is the same thing. >>> >> To be clear, I am not saying it is the same thing. What I think is >> that it would be a mistake to use a mildly unreliable heuristic by >> default (the locale + device encoding heuristic) but refuse to >> trust a more reliable heuristic (the BOM-based detection >> algorithm). >> > > I concur. On Windows both UTF-8 and signature are very common, yet > the platform default is the truly awful CP1252. While I would support combining BOM detection in the case where a file is opened for reading and no encoding is specified, I see two problems: a) if a seek operations is performed before having looked at the BOM, no determination would have been made b) what encoding should it use on writing? Regards, Martin From v+python at g.nevcal.com Sat Jan 9 02:49:12 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 17:49:12 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <4B47E098.6030703@g.nevcal.com> On approximately 1/8/2010 5:12 PM, came the following characters from the keyboard of MRAB: > Glenn Linderman wrote: >> On approximately 1/8/2010 3:59 PM, came the following characters from >> the keyboard of Victor Stinner: >>> Hi, >>> >>> Thanks for all the answers! I will try to sum up all ideas here. >> >> One concern I have with this implementation encoding="BOM" is that if >> there is no BOM it assumes UTF-8. That is probably a good assumption >> in some circumstances, but not in others. >> >> * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE >> encoded files include a BOM. It is only required that UTF-16 and >> UTF-32 (cases where the endianness is unspecified) contain a BOM. >> Hence, it might be that someone would expect a UTF-16LE (or any of >> the formats that don't require a BOM, rather than UTF-8), but be >> willing to accept any BOM-discriminated format. >> >> * Potentially, this could be expanded beyond the various Unicode >> encodings... one could envision that a program whose data files >> historically were in any particular national language locale, could >> want to be enhance to accept Unicode, and could declare that they >> will accept any BOM-discriminated format, but want to default, in the >> absence of a BOM, to the original national language locale that they >> historically accepted. That would provide a migration path for their >> old data files. >> >> So the point is, that it might be nice to have >> "BOM-otherEncodingForDefault" for each other encoding that Python >> supports. Not sure that is the right API, but I think it is >> expressive enough to handle the cases above. Whether the cases solve >> actual problems or not, I couldn't say, but they seem like reasonable >> cases. >> >> It would, of course, be nicest if OS metadata had been invented way >> back when, for all OSes, such that all text files were flagged with >> their encoding... then languages could just read the encoding and do >> the right thing! But we live in the real world, instead. >> > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? That sounds very flexible -- but in net effect it would only make illegal a subset of the BOM-containing encodings (those not listed) without making legal any additional encodings other than the non-BOM encoding. Whether prohibiting a subset of BOM-containing encodings is a useful use case, I couldn't say... but my goal would be to included as many different file encodings on input as possible: without a BOM, that is exactly 1 (unless there are other heuristics), with a BOM, it is 1+all-BOM-containing encodings. Your scheme would permit numbers of encodings accepted to vary between 1 and 1+all-BOM-containing encodings. (I think everyone can agree there are 5 different byte sequences that can be called a Unicode BOM. The likelihood of them appearing in any other text encoding created by mankind depends on those other encodings -- but it is not impossible. It is truly up to the application to decide whether BOM detection could potentially conflict with files in some other encoding that would be acceptable to the application.) So I think it is taking it further than I can see value in, but I'm willing to be convinced otherwise. I see only a need for detecting BOM, and specifying a default encoding to be used if there is no BOM. Note that it might be nice to have a specification for using current encoding=None heuristic -- perhaps encoding="BOM-None" per my originally proposed syntax. But I'm still not saying that is the best syntax. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ncoghlan at gmail.com Sat Jan 9 04:07:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 09 Jan 2010 13:07:12 +1000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B476196.4080409@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B476196.4080409@mrabarnett.plus.com> Message-ID: <4B47F2E0.7090403@gmail.com> MRAB wrote: > Maybe there should also be a way of determining what encoding it decided > it was, so that you can then write a new file in that same encoding. I thought of that question as well - the f.encoding attribute on the opened file should be sufficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Sat Jan 9 06:48:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 06:48:36 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47E098.6030703@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> <4B47E098.6030703@g.nevcal.com> Message-ID: <319e029f1001082148h5cee2c0bn76f70a17766ca69e@mail.gmail.com> It seems to me that when opening a file, the following is the only flow that makes sense for the typical opening of a file flow: if encoding is not None: use encoding elif file has BOM: use BOM else: use system default And hence a encoding='BOM' isn't needed there. Although I'm trying to come up with usecases that doesn't work with this, I can't. :) BUT When writing things are not so easy though. Apparently some encodings require a BOM to be written, but others do not, but allow it, and some has no byte order mark. So there you have to be able to write the BOM, or not. And that's either a new parameter, because you can't use encoding='BOM' since you need to specify the encoding as well, or a new method. I would suggest a BOM parameter, and maybe a method as well. BOM=None|True|False Where "None" means a sane default behaviour, that is write a BOM if the encoding require it. "True" means write a BOM if the encoding *supports* it. "False" means Don't write a BOM even if the encoding requires it (because I know what I'm doing) if 'w' in mode: # But not 'r' or 'a' if BOM == True and encoding in (ENCODINGS THAT ALLOW BOM): write_bom = True elif BOM == False: write_bom = False elif BOM == None and encoding in (ENCODINGS THAT REQUIRE BOM): write_bom = True else: write_bom = False else: write_bom = False For reading this parameter could either be a noop, or possibly change the behavior somehow, if a usecase where that makes sense can be imagined. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From walter at livinglogic.de Sat Jan 9 11:51:57 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 11:51:57 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B485FCD.2040901@livinglogic.de> On 09.01.10 01:47, Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. This is doable with the currect API. Simply define a codec search function that handles all encoding names that start with "BOM-" and pass the "otherEncodingForDefault" part along to the codec. > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. Servus, Walter From walter at livinglogic.de Sat Jan 9 12:18:33 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 12:18:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081140.28124.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> Message-ID: <4B486609.2050804@livinglogic.de> Victor Stinner wrote: > Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : >>> Builtin open() function is unable to open an UTF-16/32 file starting with >>> a BOM if the encoding is not specified (raise an unicode error). For an >>> UTF-8 file starting with a BOM, read()/readline() returns also the BOM >>> whereas the BOM should be "ignored". >> It depends. If you use the utf-8-sig encoding, it *will* ignore the >> UTF-8 signature. > > Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and > UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to > remove the BOM after the first read (much harder if you use a module like > ConfigParser or csv). > >>> Since my proposition changes the result TextIOWrapper.read()/readline() >>> for files starting with a BOM, we might introduce an option to open() to >>> enable the new behaviour. But is it really needed to keep the backward >>> compatibility? >> Absolutely. And there is no need to produce a new option, but instead >> use the existing options: define an encoding that auto-detects the >> encoding from the family of BOMs. Maybe you call it encoding="sniff". > > Good idea, I choosed open(filename, encoding="BOM"). On the surface this looks like there's an encoding named "BOM", but looking at your patch I found that the check is still done in TextIOWrapper. IMHO the best approach would to the implement a *real* codec named "BOM" (or "sniff"). This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop. To see how something like this can be done, take a look at the UTF-16 codec, that switches to bigendian or littleendian mode depending on the first read/decode call. Servus, Walter From victor.stinner at haypocalc.com Sat Jan 9 13:37:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:37:06 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47DA7B.6050505@v.loewis.de> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> <4B47DA7B.6050505@v.loewis.de> Message-ID: <201001091337.06596.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:23:07, Martin v. L?wis a ?crit : > While I would support combining BOM detection in the case where a file > is opened for reading and no encoding is specified, I see two problems: > a) if a seek operations is performed before having looked at the BOM, > no determination would have been made TextIOWrapper doesn't support seek to an arbitrary byte. It uses "cookie" which is an opaque value. Reuse a cookie from another file or an old cookie is forbidden (but it doesn't raise an error). This is not specific to the BOM checking: the problem already exist for encodings using a BOM (eg. UTF-16). > b) what encoding should it use on writing? Don't change anything to writing. With Antoince choice: open('file.txt', 'w', encoding=None) continue to use the actual heuristic (os.device_encoding() or system locale). With Guido choice, encoding="BOM": it raises an error, because BOM check is not supported when writing into a file. How could the BOM be checked when creating a new (empty) file!? -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Sat Jan 9 13:45:58 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 09 Jan 2010 13:45:58 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B487A86.4020603@egenix.com> Victor Stinner wrote: > (2) Check for a BOM while reading or detect it before? > > Everybody agree that checking BOM is an interesting option and should not be > limited to open(). > > Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file > name or a binary file-like object: it returns the encoding and seek to the > file start or just after the BOM. > > I dislike this function because it requires extra file operations (open > (optional), read() and seek()) and it doesn't work if the file is not seekable > (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to > avoid extra file operations. > > Note: I implemented the BOM check in TextIOWrapper; so it's already usable for > any file-like object. Yes, but the implementation is limited to just BOM checking and thus only supports UTF-8-SIG, UTF-16 and UTF-32. With a codecs module function we could easily extend the encoding detection to more file types, e.g. XML files, Python source code files, etc. that use other mechanisms for defining the encoding. BTW: I haven't looked at your implementation, but what happens when your BOM check fails ? Will the implementation add the already read bytes back to a buffer ? This rollback action is the only reason for needing a seekable stream in codecs.guess_stream_encoding(). Another point to consider: AFAIK, we currently have a moratorium on changes to Python builtins. How does that match up with the proposed changes ? Using a new codec like Walter suggested would move the implementation into the stdlib for which doesn't the moratorium doesn't apply. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Sat Jan 9 13:54:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:54:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <201001091354.17239.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 01:47:38, vous avez ?crit : > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. If no BOM is found, it fallback to the current heuristic: os.device_encoding() or system local. > (...) Hence, it might be that someone would expect a UTF-16LE (or any of > the formats that don't require a BOM, rather than UTF-8), but be willing > to accept any BOM-discriminated format. > (...) declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted You mean "if there is a BOM, use it, otherwise fallback to a specific charset"? How could it be declared? Maybe: open("file.txt", check_bom=True, encoding="UTF16-LE") open("file.txt", check_bom=True, encoding="latin1") About falling back to UTF-8, it would be written: open("file.txt", check_bom=True, encoding="UTF-8") As explained before, check_bom=True is only accepted for read only file mode. Well, why not. This is a third choice for my point (1) :-) It's between Guido and Antoine choice, and I like it because we can fallback to UTF-8 instead of the dummy system locale: Windows users will be happy to be able to use UTF-8 :-) I prefer to fallback to a fixed encoding then depending on the system locale. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:34:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:34:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <201001091434.17521.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:12:28, MRAB a ?crit : > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? Yes, you're taking it foo far :-) Checking BOM is reliable, whereas *guessing* the charset only using the byte stream can only be an heuristic. Guess a charset is a complex problem, they are 3rd party library to do that, like the chardet project. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:38:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:38:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B486609.2050804@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <201001091438.43576.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : > > Good idea, I choosed open(filename, encoding="BOM"). > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Why not, this is another solution to the point (2) (Check for a BOM while reading or detect it before?). Which encoding would be used if there is not BOM? UTF-8 sounds like a good choice. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:50:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:50:28 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B487A86.4020603@egenix.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B487A86.4020603@egenix.com> Message-ID: <201001091450.28497.victor.stinner@haypocalc.com> Hi, Le samedi 09 janvier 2010 13:45:58, vous avez ?crit : > > Note: I implemented the BOM check in TextIOWrapper; so it's already > > usable for any file-like object. > > Yes, but the implementation is limited to just BOM checking > and thus only supports UTF-8-SIG, UTF-16 and UTF-32. Sure, but that's already better than no BOM check :-) It looks like many people would apprecite UTF-8-SIG detection, since this encoding is common on Windows. > BTW: I haven't looked at your implementation, but what happens > when your BOM check fails ? Will the implementation add the > already read bytes back to a buffer ? My implementation is done between buffer.read() and decoder.decode(data). If there is a BOM: set the encoding and remove the BOM bytes from the byte string. Otherwise, use another algorithm to choose the encoding and leave the byte string unchanged. It can be seen as a codec: it works like UTF-16 and UTF-32 codecs ;-) > AFAIK, we currently have a moratorium on changes to Python > builtins. How does that match up with the proposed changes ? Oh yes, I forgot the moratorium. In all solutions, some of them don't change the API. Eg. Antoine proposed to leave the API unchanged: open(file) => open(file) :-) I don't know if it's compatible with the moratorium or not. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 16:05:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 15:05:45 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: Walter D?rwald livinglogic.de> writes: > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Sorry but this is missing the point. The point here is to improve the open() function. I'm sure people who know about encodings are able to install the chardet library or even whip up their own BOM detection routine... Antoine. From benjamin at python.org Sat Jan 9 18:29:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 11:29:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 Message-ID: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> On behalf of the Python development team, I'm gleeful to announce the second alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From kmtracey at gmail.com Sat Jan 9 19:48:12 2010 From: kmtracey at gmail.com (Karen Tracey) Date: Sat, 9 Jan 2010 13:48:12 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce the > second > alpha release of Python 2.7. > > Well yay. Django's test suite (1242 tests) runs with just one failure on the 2.7 alpha 2 level, and that looks to be likely due to the improved string/float rounding so not really a problem, just a difference. That's down from 104 failures and 40 errors with 2.7 alpha 1. Note on the website page http://www.python.org/download/releases/2.7/ the "Change log for this release" link is still pointing to the alpha 1 changelog. Thanks, Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jan 9 19:51:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 12:51:11 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <1afaf6161001091051kb1ba08q9b96094af16c12fa@mail.gmail.com> 2010/1/9 Karen Tracey : > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson > wrote: >> >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> > > Well yay.? Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference.? That's > down from 104 failures and 40 errors with 2.7 alpha 1. Excellent! > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Thanks. I'll fix that. > -- Regards, Benjamin From skip at pobox.com Sat Jan 9 21:00:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:00:44 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 Message-ID: <19272.57452.972234.643008@montanaro.dyndns.org> How much of the Unladen Swallow cPickle speedups have been incorporated into 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the two versions I currently care about at work - we will skip 2.5 entirely). It appears some of their speedups may have already been merged to trunk, but I'm not sure how much. If a patch to merge this to 2.7 is already under consideration I won't look at it, but I interpreted Collin Winter's response to my query on the u-s mailing list that not everything has been done yet. Thx, Skip From martin at v.loewis.de Sat Jan 9 21:09:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 09 Jan 2010 21:09:27 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <4B48E277.9010408@v.loewis.de> Antoine Pitrou wrote: > Walter D?rwald livinglogic.de> writes: >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Sorry but this is missing the point. The point here is to improve the open() > function. I'm sure people who know about encodings are able to install the > chardet library or even whip up their own BOM detection routine... How does the requirement that it be implemented as a codec miss the point? FWIW, I agree with Walter that if it is provided through the encoding= argument, it should be a codec. If it is built into the open function (for whatever reason), it must be provided by some other parameter. I do see the point that it becomes available to end users only when released as part of Python. However, this *also* means that applications won't be using it for another three years or so, since they'll have to support older Python versions as well (unless it is integrated in the case where no encoding is specified). Regards, Martin From pjenvey at underboss.org Sat Jan 9 21:09:29 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 9 Jan 2010 12:09:29 -0800 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <19272.57452.972234.643008@montanaro.dyndns.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> On Jan 9, 2010, at 12:00 PM, skip at pobox.com wrote: > How much of the Unladen Swallow cPickle speedups have been incorporated into > 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the > two versions I currently care about at work - we will skip 2.5 entirely). > It appears some of their speedups may have already been merged to trunk, but > I'm not sure how much. If a patch to merge this to 2.7 is already under > consideration I won't look at it, but I interpreted Collin Winter's response > to my query on the u-s mailing list that not everything has been done yet. They've documented their upstream patches here: http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches -- Philip Jenvey From skip at pobox.com Sat Jan 9 21:21:20 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:21:20 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> Message-ID: <19272.58688.173011.412712@montanaro.dyndns.org> Philip> They've documented their upstream patches here: Philip> http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches Thanks. That will help immensely. Skip From solipsis at pitrou.net Sat Jan 9 21:28:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:28:12 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Sorry but this is missing the point. The point here is to improve the open() > > function. I'm sure people who know about encodings are able to install the > > chardet library or even whip up their own BOM detection routine... > > How does the requirement that it be implemented as a codec miss the > point? If we want it to be the default, it must be able to fallback on the current locale-based algorithm if no BOM is found. I don't think it would be easy for a codec to do that. > FWIW, I agree with Walter that if it is provided through the encoding= > argument, it should be a codec. If it is built into the open function > (for whatever reason), it must be provided by some other parameter. Why not simply encoding=None? The default value should provide the most useful behaviour possible. Forcing users to choose between two different autodetection strategies (encoding=None and another one) is a little insane IMO. Regards Antoine. From solipsis at pitrou.net Sat Jan 9 21:32:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:32:27 +0000 (UTC) Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > If a patch to merge this to 2.7 is already under > consideration I won't look at it, Why won't you look at it? :) Actually, if these patches are to be merged someone should certainly look at them, and do the (possibly) remaining work. http://bugs.python.org/issue5683 http://bugs.python.org/issue5671 Thank you Antoine. From skip at pobox.com Sat Jan 9 21:43:42 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:43:42 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <19272.60030.25319.269597@montanaro.dyndns.org> >>>>> "Antoine" == Antoine Pitrou writes: Antoine> pobox.com> writes: >> >> If a patch to merge this to 2.7 is already under >> consideration I won't look at it, Antoine> Why won't you look at it? :) I meant I wouldn't look at developing one. Skip From regebro at gmail.com Sat Jan 9 23:14:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 23:14:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Right. It seems like encoding=None is the right way to go there. encoding='BOM' would probably only work if 'BOM' isn't an encoding but a special tag, which is ugly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Sun Jan 10 00:25:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 23:25:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> Message-ID: <4B49105E.303@voidspace.org.uk> On 09/01/2010 22:14, Lennart Regebro wrote: > On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. >> > Right. It seems like encoding=None is the right way to go there. > encoding='BOM' would probably only work if 'BOM' isn't an encoding but > a special tag, which is ugly. > > I would rather see it as the default behavior for open without an encoding specified. I know Guido has expressed a preference against this so I won't continue to flog it. The current behavior however is that we have a 'guessing' algorithm based on the platform default. Currently if you open a text file in read mode that has a UTF-8 signature, but the platform default is something other than UTF-8, then we open the file using what is likely to be the incorrect encoding. Looking for the signature seems to be better behaviour in that case. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jan 10 00:40:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 10 Jan 2010 00:40:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <4B4913DF.7050801@v.loewis.de> >> How does the requirement that it be implemented as a codec miss the >> point? > > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Yes - however, Victor currently apparently *doesn't* want it to be the default, but wants the user to specify encoding="BOM". If so, it isn't the default, and it is easy to implement as a codec. >> FWIW, I agree with Walter that if it is provided through the encoding= >> argument, it should be a codec. If it is built into the open function >> (for whatever reason), it must be provided by some other parameter. > > Why not simply encoding=None? I don't mind. Please re-read Walter's message - it only said that *if* this is activated through encoding="BOM", *then* it must be a codec, and could be on PyPI. I don't think Walter was talking about the case "it is not activated through encoding='BOM'" *at all*. > The default value should provide the most useful > behaviour possible. Forcing users to choose between two different autodetection > strategies (encoding=None and another one) is a little insane IMO. That wouldn't disturb me much. There are a lot of things in that area that are a little insane, starting with Microsoft Windows :-) Regards, Martin From henning.vonbargen at arcor.de Sun Jan 10 12:10:02 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Sun, 10 Jan 2010 12:10:02 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: Message-ID: <4B49B58A.4040103@arcor.de> If Python should support BOM when reading text files, it should also be able to *write* such files. An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what? That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values: None: default (old) behaviour: don't handle BOM at all True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument. (*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False. Henning P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway... From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brett at python.org Sun Jan 10 19:51:26 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 10:51:26 -0800 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: Nick Coghlan thought I should forward this to python-dev so people are aware of this change and why it occurred (plus it indirectly informs Guido I finally finished the work). ---------- Forwarded message ---------- From: brett.cannon Date: Sat, Jan 9, 2010 at 18:56 Subject: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c To: python-checkins at python.org Author: brett.cannon Date: Sun Jan 10 03:56:19 2010 New Revision: 77402 Log: DeprecationWarning is now silent by default. This was originally suggested by Guido, discussed on the stdlib-sig mailing list, and given the OK by Guido directly to me. What this change essentially means is that Python has taken a policy of silencing warnings that are only of interest to developers by default. This should prevent users from seeing warnings which are triggered by an application being run against a new interpreter before the app developer has a chance to update their code. Closes issue #7319. Thanks to Antoine Pitrou, Ezio Melotti, and Brian Curtin for helping with the issue. Modified: python/trunk/Doc/library/warnings.rst python/trunk/Lib/test/test_ascii_formatd.py python/trunk/Lib/test/test_exceptions.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS python/trunk/Python/_warnings.c Modified: python/trunk/Doc/library/warnings.rst ============================================================================== --- python/trunk/Doc/library/warnings.rst (original) +++ python/trunk/Doc/library/warnings.rst Sun Jan 10 03:56:19 2010 @@ -59,7 +59,7 @@ | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,14 +151,6 @@ :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - .. _warning-suppress: @@ -226,6 +221,37 @@ entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions Modified: python/trunk/Lib/test/test_ascii_formatd.py ============================================================================== --- python/trunk/Lib/test/test_ascii_formatd.py (original) +++ python/trunk/Lib/test/test_ascii_formatd.py Sun Jan 10 03:56:19 2010 @@ -4,6 +4,7 @@ import unittest from test_support import check_warnings, run_unittest, cpython_only +import warnings class FormatDeprecationTests(unittest.TestCase): @@ -17,6 +18,7 @@ buf = create_string_buffer(' ' * 100) with check_warnings() as w: + warnings.simplefilter('default') PyOS_ascii_formatd(byref(buf), sizeof(buf), '%+.10f', c_double(10.0)) self.assertEqual(buf.value, '+10.0000000000') Modified: python/trunk/Lib/test/test_exceptions.py ============================================================================== --- python/trunk/Lib/test/test_exceptions.py (original) +++ python/trunk/Lib/test/test_exceptions.py Sun Jan 10 03:56:19 2010 @@ -309,6 +309,7 @@ # BaseException.__init__ triggers a deprecation warning. exc = BaseException("foo") with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('default') self.assertEquals(exc.message, "foo") self.assertEquals(len(w), 1) self.assertEquals(w[0].category, DeprecationWarning) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Sun Jan 10 03:56:19 2010 @@ -383,8 +383,8 @@ # Module initialization _processoptions(sys.warnoptions) if not _warnings_defaults: - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) + for cls in (DeprecationWarning, PendingDeprecationWarning, ImportWarning): + simplefilter("ignore", category=cls, append=True) bytes_warning = sys.flags.bytes_warning if bytes_warning > 1: bytes_action = "error" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Jan 10 03:56:19 2010 @@ -12,6 +12,8 @@ Core and Builtins ----------------- +- Issue #7319: Silence DeprecationWarning by default. + - Issue #2335: Backport set literals syntax from Python 3.x. Library Modified: python/trunk/Python/_warnings.c ============================================================================== --- python/trunk/Python/_warnings.c (original) +++ python/trunk/Python/_warnings.c Sun Jan 10 03:56:19 2010 @@ -85,10 +85,10 @@ default_action = get_warnings_attr("defaultaction"); if (default_action == NULL) { - if (PyErr_Occurred()) { - return NULL; - } - return _default_action; + if (PyErr_Occurred()) { + return NULL; + } + return _default_action; } Py_DECREF(_default_action); @@ -202,12 +202,12 @@ mod_str = PyString_AsString(filename); if (mod_str == NULL) - return NULL; + return NULL; len = PyString_Size(filename); if (len < 0) return NULL; if (len >= 3 && - strncmp(mod_str + (len - 3), ".py", 3) == 0) { + strncmp(mod_str + (len - 3), ".py", 3) == 0) { module = PyString_FromStringAndSize(mod_str, len-3); } else { @@ -251,7 +251,7 @@ name = PyObject_GetAttrString(category, "__name__"); if (name == NULL) /* XXX Can an object lack a '__name__' attribute? */ - return; + return; f_stderr = PySys_GetObject("stderr"); if (f_stderr == NULL) { @@ -341,7 +341,7 @@ rc = already_warned(registry, key, 0); if (rc == -1) goto cleanup; - else if (rc == 1) + else if (rc == 1) goto return_none; /* Else this warning hasn't been generated before. */ } @@ -492,9 +492,9 @@ /* Setup filename. */ *filename = PyDict_GetItemString(globals, "__file__"); if (*filename != NULL) { - Py_ssize_t len = PyString_Size(*filename); + Py_ssize_t len = PyString_Size(*filename); const char *file_str = PyString_AsString(*filename); - if (file_str == NULL || (len < 0 && PyErr_Occurred())) + if (file_str == NULL || (len < 0 && PyErr_Occurred())) goto handle_error; /* if filename.lower().endswith((".pyc", ".pyo")): */ @@ -506,10 +506,10 @@ tolower(file_str[len-1]) == 'o')) { *filename = PyString_FromStringAndSize(file_str, len-1); - if (*filename == NULL) - goto handle_error; - } - else + if (*filename == NULL) + goto handle_error; + } + else Py_INCREF(*filename); } else { @@ -536,8 +536,8 @@ else { /* embedded interpreters don't have sys.argv, see bug #839151 */ *filename = PyString_FromString("__main__"); - if (*filename == NULL) - goto handle_error; + if (*filename == NULL) + goto handle_error; } } if (*filename == NULL) { @@ -839,26 +839,29 @@ static PyObject * init_filters(void) { - PyObject *filters = PyList_New(3); + PyObject *filters = PyList_New(4); const char *bytes_action; if (filters == NULL) return NULL; PyList_SET_ITEM(filters, 0, + create_filter(PyExc_DeprecationWarning, "ignore")); + PyList_SET_ITEM(filters, 1, create_filter(PyExc_PendingDeprecationWarning, "ignore")); - PyList_SET_ITEM(filters, 1, create_filter(PyExc_ImportWarning, "ignore")); + PyList_SET_ITEM(filters, 2, create_filter(PyExc_ImportWarning, "ignore")); if (Py_BytesWarningFlag > 1) bytes_action = "error"; else if (Py_BytesWarningFlag) bytes_action = "default"; else bytes_action = "ignore"; - PyList_SET_ITEM(filters, 2, create_filter(PyExc_BytesWarning, + PyList_SET_ITEM(filters, 3, create_filter(PyExc_BytesWarning, bytes_action)); if (PyList_GET_ITEM(filters, 0) == NULL || PyList_GET_ITEM(filters, 1) == NULL || - PyList_GET_ITEM(filters, 2) == NULL) { + PyList_GET_ITEM(filters, 2) == NULL || + PyList_GET_ITEM(filters, 3) == NULL) { Py_DECREF(filters); return NULL; } _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sun Jan 10 20:22:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 10 Jan 2010 20:22:10 +0100 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: <201001102022.10259.victor.stinner@haypocalc.com> Hi, Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : > Nick Coghlan thought I should forward this to python-dev so people are > aware of this change and why it occurred (plus it indirectly informs Guido > I finally finished the work). Thanks to copy this mail to the python-dev mailing list. What is the recommanded way to get the previous behaviour (print deprecation warnings once)? Is there a command line option? Is it "-W default"? -- Victor Stinner http://www.haypocalc.com/ From benjamin at python.org Sun Jan 10 20:23:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 13:23:54 -0600 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <201001102022.10259.victor.stinner@haypocalc.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> <201001102022.10259.victor.stinner@haypocalc.com> Message-ID: <1afaf6161001101123g366d80dbjeaa80f1a632605e2@mail.gmail.com> 2010/1/10 Victor Stinner : > Hi, > > Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : >> Nick Coghlan thought I should forward this to python-dev so people are >> ?aware of this change and why it occurred (plus it indirectly informs Guido >> ?I finally finished the work). > > Thanks to copy this mail to the python-dev mailing list. > > What is the recommanded way to get the previous behaviour (print deprecation > warnings once)? Is there a command line option? Is it "-W default"? That commit conveniently adds documentation answering that question. :) -- Regards, Benjamin From nas at arctrix.com Sun Jan 10 20:30:09 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:30:09 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce > the second alpha release of Python 2.7. Thanks to everyone who contributed. > Python 2.7 is scheduled to be the last major version in the 2.x > series. Has this really been decided already? Maybe I missed it. In my opinion, it does Python's reputation harm to make such a statement. Conservative users (probably the vast majority of Python users) don't like to hear that software they are considering using is nearing the end of its life. What does it gain us to announce that the 2.x branch is dead aside from bugfixes? I propose that the 2.x branch be treated like 2.x.y branches: as long as there is sufficient volunteer labour, it should continue to live. In order to avoid wasted development effort, it would be prudent to announce that unless a 2.8 release manager steps up, whatever is committed to the 2.x branch after the 2.7 release may never get released. Said another way, it's okay for the Python developers to decide to abandon 2.x and put their efforts into 3.x. It's not okay for them to prevent others from continuing to work on 2.x or to somehow make 2.x look worse so 3.x looks better. Python 3 needs to stand on its own terms and I'm confident it can. Best regards, Neil From brett at python.org Sun Jan 10 21:09:08 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:09:08 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sun, Jan 10, 2010 at 11:30, Neil Schemenauer wrote: > Benjamin Peterson wrote: > > On behalf of the Python development team, I'm gleeful to announce > > the second alpha release of Python 2.7. > > Thanks to everyone who contributed. > > > Python 2.7 is scheduled to be the last major version in the 2.x > > series. > > Has this really been decided already? Maybe I missed it. More or less. It was first discussed at the language summit last year and has come up here a couple of times. If needed we can make it official in terms of lifetime of 2.7, etc. at the language summit this year. > In my > opinion, it does Python's reputation harm to make such a statement. > Conservative users (probably the vast majority of Python users) > don't like to hear that software they are considering using is > nearing the end of its life. What does it gain us to announce that > the 2.x branch is dead aside from bugfixes? > > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. > > Said another way, it's okay for the Python developers to decide to > abandon 2.x and put their efforts into 3.x. It's not okay for them > to prevent others from continuing to work on 2.x or to somehow make > 2.x look worse so 3.x looks better. Python 3 needs to stand on its > own terms and I'm confident it can. > I don't think ending the 2.x series at 2.7 makes it look bad compared to 3.2; it's simply the end of a development line like any other software project. I suspect 2.7 will have a protracted bugfix window because so much code runs on 2.x exclusively at the moment. And if core developers want to continue to backport fixes past two years from release they can (or however long we decide to officially support 2.7). No one is saying people still can't work on the code, just that python-dev as an entity is not going to focus its effort on the 2.x series anymore and people should not rely upon us to continue to focus new development efforts in that series. If there are core developers who want to continue to do bugfix releases then that's fine and I am happy to flag patches as needing backports and let other's do the work after the standard two year maintenance cycle, but I know I do not want to be held accountable as a core developer to keep the 2.x going indefinitely. Maintaining four branches is enough of a reason in my book to not keep the 2.x series going. If there really is an outcry on this we can re-visit the issue, but as of right now we need to move forward at some point and 2.7 seems like that good point. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 10 21:23:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Jan 2010 06:23:27 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A373F.9050601@gmail.com> Neil Schemenauer wrote: > In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. The announcement is precisely to avoid the situation where people commit new features to the 2.x main line of development (after the 2.7 maintenance branch is created) in the expectation that they will be released as part of a hypothetical 2.8 release. Whether that info needs to be in each and every 2.7 announcement... probably not. It isn't really info for users of Python, just for developers of Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jan 10 21:25:29 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:25:29 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit Message-ID: Michael has given me the hg transition/stdlib time slot at the language summit this year. In regards to that I plan to lead a discussion on: * where we are at w/ the Hg transition (Dirkjan should be there and I did a blog post on this topic recently: http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) * argparse (PEP 389) * brief mention on still wanting to break out the stdlib from CPython * an official policy on extension modules? (i.e. must have a pure Python implementation which can mean a ctypes implementation unless given an explicit waiver) That's everything from a stdlib perspective. I have half-baked ideas, but nothing concrete enough to bring up unless people really want to discuss stuff like how to potentially standardize module deprecation warnings, etc. In terms of me personally, I do plan to bring up at some point during the summit these points which don't squarely fit during my time slot: * an official unofficial policy on how new proposed features should come to us (i.e. working code to python-ideas with a shell of a PEP that includes abstract and motivation -> hashed out PEP to python-dev -> pronouncement) * any changes needed to the issue tracker to help with the workflow? (stage field seems like a failed experiment and we now have several effective triage people who can help w/ guiding changes) If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Sun Jan 10 22:52:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 10 Jan 2010 22:52:00 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) Unfortunately my flight doesn't land until about the time the Mercurial slot ends, so while I'll be there later on that afternoon for sprinting or questions or anything, I won't be at the actual Mercurial talk at the summit. Cheers, Dirkjan From fuzzyman at voidspace.org.uk Sun Jan 10 23:27:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 10 Jan 2010 22:27:58 +0000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4A546E.8010808@voidspace.org.uk> On 10/01/2010 21:52, Dirkjan Ochtman wrote: > On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > >> Michael has given me the hg transition/stdlib time slot at the language >> summit this year. In regards to that I plan to lead a discussion on: >> * where we are at w/ the Hg transition (Dirkjan should be there and I did a >> blog post on this topic recently: >> http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) >> > Unfortunately my flight doesn't land until about the time the > Mercurial slot ends, so while I'll be there later on that afternoon > for sprinting or questions or anything, I won't be at the actual > Mercurial talk at the summit. > > We will probably be in 'open discussion' when you arrive so it will still be good to hear from you on the topic and there maybe questions that can't be answered until you arrive... :-) Michael > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Mon Jan 11 00:02:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:02:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A5C69.7090007@v.loewis.de> > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. I think it's more difficult than that. It also takes developer effort to *commit* to the trunk. So if the trunk continued to live, would it still be ok if I closed a 2.x feature request as "won't fix", or only committed the new feature to the 3.x branch? As for old branches: they *don't* live in the way you claim (i.e. remain open with changes potentially just not released). Instead, at some point, they are frozen to bug-fix only, then to security-fix only, and then to no change at all. Regards, Martin From martin at v.loewis.de Mon Jan 11 00:07:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:07:58 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A373F.9050601@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> Message-ID: <4B4A5DCE.3070909@v.loewis.de> > The announcement is precisely to avoid the situation where people commit > new features to the 2.x main line of development (after the 2.7 > maintenance branch is created) in the expectation that they will be > released as part of a hypothetical 2.8 release. > > Whether that info needs to be in each and every 2.7 announcement... > probably not. It isn't really info for users of Python, just for > developers of Python. I think the announcement is indeed also for the users, and I would prefer it to stay in the release announcements, so that people will know for sure (and speak up) before the 2.7 release happens. As for decisions: I don't think there was an official BDFL pronouncent, but I recall Guido posting a message close to that, proposing that 2.7 will be a release that will see bug fix releases for an indefinite period of time (where indefinite != infinite). This was shortly after him proposing that perhaps we shouldn't make a 2.7 release at all, and stop at 2.6. As for such a decision giving a bad light on Python: I don't think that will be the case. Instead, I hear many users surprised for how long we have been maintaining to parallel versions - "that must have taken a lot of effort". So everybody will likely understand that enough is enough. Regards, Martin From benjamin at python.org Mon Jan 11 01:07:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 18:07:35 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Consider this program: class Descr(object): def __init__(self, name): self.name = name def __set__(self, instance, what): instance.__dict__[self.name] = what class X(object): attr = Descr("attr") x = X() print(x.attr) x.attr = 42 print(x.attr) It gives in output: <__main__.Descr object at 0x7fe1c9b28150> 42 The documentation [1] says that Descr is a data descriptor because it defines the __set__ method. It also states that data descriptors always override the value in the instance dictionary. So, the second line should also be the descriptor object according to the documentation. My question is: Is this a doc bug or a implementation bug? If the former, it will be the description of a data descriptor much less consistent, since it will require that a __get__ method be present, too. If the latter, the fix may break some programs relying on the ability to "cache" a value in the instance dictionary. [1] http://docs.python.org/reference/datamodel#invoking-descriptors -- Regards, Benjamin From amauryfa at gmail.com Mon Jan 11 01:51:09 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Jan 2010 01:51:09 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: Hi, 2010/1/11 Benjamin Peterson : > Consider this program: > > class Descr(object): > ? ?def __init__(self, name): > ? ? ? ?self.name = name > ? ?def __set__(self, instance, what): > ? ? ? ?instance.__dict__[self.name] = what > > class X(object): > ? ?attr = Descr("attr") > > x = X() > print(x.attr) > x.attr = 42 > print(x.attr) > > It gives in output: > > <__main__.Descr object at 0x7fe1c9b28150> > 42 > > The documentation [1] says that Descr is a data descriptor because it > defines the __set__ method. It also states that data descriptors > always override the value in the instance dictionary. So, the second > line should also be the descriptor object according to the > documentation. > > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors Quoting the documentation: """Normally, data descriptors define both __get__() and __set__(), while non-data descriptors have just the __get__() method. """ Your example is neither a data descriptor nor a non-data descriptor... The thing that worries me a bit is the "x.attr" returning the Descr object. Descriptors should remain at the class level, and instance should only see values. I'd prefer an AttributeError in this case. -- Amaury Forgeot d'Arc From benjamin at python.org Mon Jan 11 02:00:32 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 19:00:32 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <1afaf6161001101700u21006708l2ef62bfa257e4ce7@mail.gmail.com> 2010/1/10 Amaury Forgeot d'Arc : > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... See the footnote: http://docs.python.org/reference/datamodel#id7 > > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. Far too late for that, I'm afraid. -- Regards, Benjamin From nas at arctrix.com Mon Jan 11 02:44:57 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:44:57 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <20100111014457.GA5289@arctrix.com> On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > I don't think ending the 2.x series at 2.7 makes it look bad > compared to 3.2; it's simply the end of a development line like > any other software project. I suspect 2.7 will have a protracted > bugfix window because so much code runs on 2.x exclusively at the > moment. I would guess over 99% of all Python code written doesn't run on Python 3. Given that, I think it is premature to close the door on new major versions of Python 2.x. Also, we as a project should be careful not to present the image that Python 2.x will not be supported in the future. > If there really is an outcry on this we can re-visit the issue, > but as of right now we need to move forward at some point and 2.7 > seems like that good point. I think that's bad PR. If I had a successful product, I would not announce its end of life just to see how many customers scream and then decide if I should devote more resources to continue maintaining it. IMHO, the release notes should say something like: After the Python 2.7 release, the focus of Python development will be on Python 3. There will continue to be maintainance releases of Python 2.x. trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil From tjreedy at udel.edu Mon Jan 11 03:26:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Jan 2010 21:26:34 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On 1/10/2010 8:44 PM, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: >> I don't think ending the 2.x series at 2.7 makes it look bad >> compared to 3.2; it's simply the end of a development line like >> any other software project. I suspect 2.7 will have a protracted >> bugfix window because so much code runs on 2.x exclusively at the >> moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. If the removal of old features had been done in the 2.x series, as once planned (Guido originally proposed removing the old meaning of int / int in 2.5) the same more or less would be true of 2.7. It is past time for other old and now duplicated features to be removed also. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > >> If there really is an outcry on this we can re-visit the issue, >> but as of right now we need to move forward at some point and 2.7 >> seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. Python is not being ended, but upgraded (with bloating duplications removed). Think of 3.1 as 2.8 with a new name. tjr From lrekucki at gmail.com Mon Jan 11 04:26:48 2010 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Mon, 11 Jan 2010 04:26:48 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: > Date: Mon, 11 Jan 2010 01:51:09 +0100 > From: "Amaury Forgeot d'Arc" > To: Benjamin Peterson > Cc: Python Dev > Subject: Re: [Python-Dev] Data descriptor doc/implementation > ? ? ? ?inconsistency > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > 2010/1/11 Benjamin Peterson : >> Consider this program: >> >> class Descr(object): >> ? ?def __init__(self, name): >> ? ? ? ?self.name = name >> ? ?def __set__(self, instance, what): >> ? ? ? ?instance.__dict__[self.name] = what >> >> class X(object): >> ? ?attr = Descr("attr") >> >> x = X() >> print(x.attr) >> x.attr = 42 >> print(x.attr) >> >> It gives in output: >> >> <__main__.Descr object at 0x7fe1c9b28150> >> 42 >> >> The documentation [1] says that Descr is a data descriptor because it >> defines the __set__ method. It also states that data descriptors >> always override the value in the instance dictionary. So, the second >> line should also be the descriptor object according to the >> documentation. >> >> My question is: Is this a doc bug or a implementation bug? If the >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors > > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... Actually, there is this footnote [1]: """ A descriptor can define any combination of __get__(), __set__() and __delete__(). If it does not define __get__(), then accessing the attribute even on an instance will return the descriptor object itself. If the descriptor defines __set__() and/or __delete__(), it is a data descriptor; if it defines neither, it is a non-data descriptor. """ Which would mean Descr is actually a data descriptor without a __get__(), so x.attr should always return the descriptor object itself (at least in the docs). > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. > > -- > Amaury Forgeot d'Arc [1] http://docs.python.org/reference/datamodel#id7 -- Lukasz Rekucki From brett at python.org Mon Jan 11 04:46:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 19:46:04 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > > I don't think ending the 2.x series at 2.7 makes it look bad > > compared to 3.2; it's simply the end of a development line like > > any other software project. I suspect 2.7 will have a protracted > > bugfix window because so much code runs on 2.x exclusively at the > > moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Well yeah; Python 3.0 is just over a year old with 3.1 -- the first robust 3.x release - is about six months old. From the beginning uptake was expected to take years, not months, so saying that 3.x is not popular enough seems premature. > Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > No one has said bugfixes will cease. > > > If there really is an outcry on this we can re-visit the issue, > > but as of right now we need to move forward at some point and 2.7 > > seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. > > I never said that I wanted to make this announcement specifically to provoke outcries. I said if there happened to be outcries we could possibly visit the issue again. Basically I was being considerate and trying to leave the door open to discuss things in the future even though I don't see the situation changing. > IMHO, the release notes should say something like: > > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. > No because that suggests new features will be coming to 2.x which is not going to happen. If you want to say there will be continual bugfix releases for 2.7 as is par the course for Python and that the number of bugfix releases might be more than normal then I am okay with that. > > > trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil > That came and went already a couple months ago when we discussed stopping at 2.6 instead of 2.7 (see the various threads at http://mail.python.org/pipermail/python-dev/2009-November/thread.html). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 05:05:07 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:05:07 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <20100111040507.GA7200@arctrix.com> On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > After the Python 2.7 release, the focus of Python development > > will be on Python 3. There will continue to be maintainance > > releases of Python 2.x. > > No because that suggests new features will be coming to 2.x which is not > going to happen. If you want to say there will be continual bugfix releases > for 2.7 as is par the course for Python and that the number of bugfix > releases might be more than normal then I am okay with that. Are you are saying that if someone steps up to merge the Unladen Swallow features into a 2.8 release and someone also steps up to cut the release that they will be prevented from doing so? Also, are you presuming to channel the BDFL or was this dicussed somewhere other than python-dev? Maybe I'm overreacting but I get the feeling that the larger and less active segment of the Python develpment team hasn't been involved in these decisions. Best regards, Neil From nas at arctrix.com Mon Jan 11 05:17:54 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:17:54 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5C69.7090007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> Message-ID: <20100111041754.GB7200@arctrix.com> On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: > [...] would it still be ok if I closed a 2.x feature request as > "won't fix", or only committed the new feature to the 3.x branch? Yes. Non-bugfix development on 2.x would optional (i.e. done by people who want to spend the time). Since language changes are now out (an initiative I completely support), the majority of non-bugfix changes would be optimizations and platform porting. Regards, Neil From brett at python.org Mon Jan 11 06:06:15 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 21:06:15 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111040507.GA7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 20:05, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > > After the Python 2.7 release, the focus of Python development > > > will be on Python 3. There will continue to be maintainance > > > releases of Python 2.x. > > > > No because that suggests new features will be coming to 2.x which is not > > going to happen. If you want to say there will be continual bugfix > releases > > for 2.7 as is par the course for Python and that the number of bugfix > > releases might be more than normal then I am okay with that. > > Are you are saying that if someone steps up to merge the Unladen > Swallow features into a 2.8 release and someone also steps up to cut > the release that they will be prevented from doing so? > > I honestly don't know, but it's a possibility just like any other new feature request. If people start taking the carrots we have added to 3.x and backporting them to keep the 2.x series alive you are essentially making the 3.x DOA by negating its benefits which I personally don't agree with. > Also, are you presuming to channel the BDFL or was this dicussed > somewhere other than python-dev? This was discussed; see the November 2009 threads. Guido actually suggested ending the 2.x branch at 2.6 until people spoke up about the amount of stuff already backported to 2.7 from 3.x because it was being assumed to be the end of the series to warrant keeping it to help transition to 2.7. > Maybe I'm overreacting but I get > the feeling that the larger and less active segment of the Python > develpment team hasn't been involved in these decisions. > This all came up in November from the 3rd through the 6th (four days) over a ton of email traffic. This was not a snap decision but a heated discussion where even Guido participated that culminated with the decision to stop at 2.7 for the 2.x series; this was not a smoked-filled room decision. I'm sorry if people missed it when they weren't looking, but python-dev is supposed to be the place to watch for this kind of stuff. I don't know how else to bring this stuff to people's attention short of also on python-committers, but developers are basically expected to be on both lists so I don't see where anyone did anything wrong in terms of informing developers. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 06:27:13 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 23:27:13 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: <20100111052713.GA8211@arctrix.com> On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > If people start taking the carrots we have added to 3.x and > backporting them to keep the 2.x series alive you are essentially > making the 3.x DOA by negating its benefits which I personally > don't agree with. I think we have got to the heart of our disagreement. Assume that some superhuman takes all the backwards compatible goodies from 3.x and merges them into 2.x. If that really would kill off Python 3 then I would proclaim it a project that deserved to die. Why should the backwards incompatible changes be endured if they cannot justify their existance by the benefit they provide? I guess I have more confidence in Python 3 than you do. I don't see why Python 2.x needs to be artificially limited so that Python 3 can benefit. Regards, Neil From brett at python.org Mon Jan 11 07:30:49 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 22:30:49 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 21:27, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. > > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. If that really would kill off Python 3 > then I would proclaim it a project that deserved to die. Why should > the backwards incompatible changes be endured if they cannot justify > their existance by the benefit they provide? > > When I said "carrots" I meant everything that makes Python 3 the best version of Python, including the backward-incompatible changes. I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. > I don't view it as artificial but simply a focusing of the development team on what has been determined as the future of Python by Guido and letting go of the past. IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying "Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out". That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Jan 11 08:08:14 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 08:08:14 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <319e029f1001102308p5de87cber2131f3aec1abc9f0@mail.gmail.com> On Mon, Jan 11, 2010 at 06:27, Neil Schemenauer wrote: > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. The superhumans of core developers pretty much already have. That's pretty much 2.7 you are talking about. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From chrism at plope.com Mon Jan 11 08:27:03 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 11 Jan 2010 02:27:03 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4AD2C7.1050703@plope.com> Brett Cannon wrote: > IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev > saying "Python 3 is the future, but we are keeping the old Python 2.x > around because we don't have *that* much faith in the future we have > laid out". That's poison to Python 3 by showing a lack of confidence in > the direction that the BDFL and python-dev as a group has chosen. Now I > could be wrong and there could actually be a large number of active > contributors who want to keep the 2.x series going, but based on the > discussion that occurred the last time this came up I believe the guys > who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort "Python 2". - C From martin at v.loewis.de Mon Jan 11 09:06:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:06:16 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: <4B4ADBF8.3030803@v.loewis.de> Neil Schemenauer wrote: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). I think *that* would give a very bad impression of Python. Depending whom you deal with, the new feature you want may or may not get added to the code base. Contributors would feel even more stranded than they do now, where it may take several years to get a patch reviewed, as you then could submit a patch, and pray that a comitter reviews it who believes in future 2.x releases. The point of setting policies is that it gives every user (contributors, committers, and "end-user" developers) a reliable foundation for expectations. Regards, Martin From arcriley at gmail.com Mon Jan 11 09:06:10 2010 From: arcriley at gmail.com (Arc Riley) Date: Mon, 11 Jan 2010 03:06:10 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AD2C7.1050703@plope.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4AD2C7.1050703@plope.com> Message-ID: after all these years, some people still run AmigaOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 11 09:11:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:11:25 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <4B4ADD2D.6070906@v.loewis.de> > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. Why that? It is a fact: 2.x will not be supported, in some future. Should we lie to users? > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. It shouldn't say that, because it wouldn't be true. Regards, Martin From martin at v.loewis.de Mon Jan 11 09:18:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:18:30 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4ADED6.5080207@v.loewis.de> > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. Because it takes too much time. Too much of my time, but apparently also too much of other people's time. Of course, the less active fraction of Python contributors may not notice, since they just chose to not contribute (which, of course, is fine). However, asking me to work twice as much as I want to on the project to keep two branches alive is just unfair. This has nothing to do with pushing 3.x, but all with managing available manpower and still providing quality software. Regards, Martin From regebro at gmail.com Mon Jan 11 09:42:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 09:42:51 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> This is what it says now: "Python 2.7 is scheduled to be the last major version in the 2.x series before it moves into 5 years of bugfix-only mode. " And during this discussion it has been noted that others than the core python team might pick up Python 2 and make releases. So maybe we can and this discussion by changing that line in future releases to be: "Python 2.7 is scheduled to be the last major version in the 2.x series released by the Python Software Foundation before it moves into 5 years of bugfix-only mode. " That doesn't exclude *others* making a Python 2.8 that could include all sorts of crazy features. :) This is mainly just to get the pointless discussion over with. The current Python core team don't want to make a 2.8, so that will not happen. Someone else might, but as Chris points out, they won't step up until they have to, which is probably at least two years from now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 10:06:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 10:06:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> Message-ID: <4B4AEA25.7010100@v.loewis.de> > "Python 2.7 is scheduled to be the last major version in the 2.x > series released by the Python Software Foundation before it moves into > 5 years of bugfix-only mode. " > > That doesn't exclude *others* making a Python 2.8 that could include > all sorts of crazy features. :) > > This is mainly just to get the pointless discussion over with. I know you are just looking for a compromise, but this shouldn't be it: the PSF has deliberately stayed out of the actual Python engineering, so the release that Benjamin makes is not done by the PSF (but by Benjamin and his contributors :-). I like the wording as it is (although I find the promise of five years of bug fix releases a bit too strong). Regards, Martin From regebro at gmail.com Mon Jan 11 10:19:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 10:19:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AEA25.7010100@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> Message-ID: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" wrote: > I know you are just looking for a compromise, but this shouldn't be it: > the PSF has deliberately stayed out of the actual Python engineering, > so the release that Benjamin makes is not done by the PSF (but by > Benjamin and his contributors :-). Hm. Yeah. That's right of course. I started with saying "official", but then I thought "official according to who?". :) So I changed it to mentioning PSF, but that doesn't work either. I guess the current writing as as good as it gets, unless we change "scheduled" to "expected" or something. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Mon Jan 11 10:42:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:42:05 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: Neil Schemenauer, 11.01.2010 05:17: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). Note that there's also the time it takes to make a release (usually involving at least one beta release), plus the possibility of introducing bugs while adding new features or even while fixing agreed bugs. All of this takes time in addition to the time people might want to invest in 'real' development on the 2.x trunk. Stefan From stephen at xemacs.org Mon Jan 11 10:59:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Jan 2010 18:59:57 +0900 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <8763793qsi.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Schemenauer writes: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. Well, I think it's *worse* than that, and I don't think you really mean "DOA", anyway. (Feel free to correct me, of course.) The problem I see with backporting lots of stuff, and/or adding new features that aren't in 3.0, to 2.x is that it will make 2.x even cruftier, when it was already crufty enough that Guido (and almost all of python-dev) bit the bullet and said "backward compatibility is no excuse for keeping something in 3.0". That surely means that a lot of python-dev denizens will declare non-support 2.x for x > 7. It's not going to be the gradual migration we've seen over the past few months as active people start to spend more and more time on 3 vs. 2; it will be a watershed. Especially if these are new features merged from outside that the "small active segment" doesn't know anything about. From the users' point of view, that amounts to a *fork*, even if it's internal and "friendly". > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. Isn't that a bit ridiculous? I just don't see any evidence that anything like that is going to happen. Worse, if we *assume* it will happen, I don't see any way to assess whether (1) Python 3 goes belly up, (2) there's an effective fork confusing the users and draining the energy of python-dev, or (3) everybody goes "wow!" because it's so cool that everybody wants to keep maintaining an extra 3 branches indefinitely. My opinion is that given the clear direction the "small active segment" is going, telling the users anything but what Brett proposed is disinformation. > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. It's not for Python 3, which you, I, and I'm pretty sure Brett-in-his- heart-of-hearts agree can take care of itself because it is *better* than Python 2. It's for Python, and for the Python community. From walter at livinglogic.de Mon Jan 11 11:37:56 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 11:37:56 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001091438.43576.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> Message-ID: <4B4AFF84.7070206@livinglogic.de> On 09.01.10 14:38, Victor Stinner wrote: > Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : >>> Good idea, I choosed open(filename, encoding="BOM"). >> >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Why not, this is another solution to the point (2) (Check for a BOM while > reading or detect it before?). Which encoding would be used if there is not > BOM? UTF-8 sounds like a good choice. UTF-8 might be a good choice, are the failback could be specified in the encoding name, i.e. open("file.txt", encoding="BOM-UTF-8") falls back to UTF-8, if there's no BOM at the start. This could be implemented via a custom codec search function (see http://docs.python.org/library/codecs.html#codecs.register for more info). Servus, Walter From regebro at gmail.com Mon Jan 11 12:12:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:12:20 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4AFF84.7070206@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> Message-ID: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: > UTF-8 might be a good choice No, fallback if there is no BOM should be the local settings, just as fallback is today if you don't specify a codec. I mean, what if you want to look for a BOM but fall back to something else? How far will we go with encoding special information in the codecs names? codec='BOM else UTF-16 else locale'? :-) BOM is not a locale, and should not be a locale. Having a locale called BOM is wrong per se. It should either be default to look for a BOM when codec=None, or a special parameter. If none of these are desired, then we need a special function that takes a filename or file handle, and looks for a BOM and returns the codec found or None. But I find that much less natural and obvious than checking the BOM when codec=None. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Mon Jan 11 12:13:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:13:00 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <319e029f1001110313u1d839deodfe06a6c7ec423cf@mail.gmail.com> On Mon, Jan 11, 2010 at 12:12, Lennart Regebro wrote: > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. D'oh! I mean codec here obviously. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 13:29:04 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 13:29:04 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4913DF.7050801@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> Message-ID: <4B4B1990.90705@livinglogic.de> On 10.01.10 00:40, "Martin v. L?wis" wrote: >>> How does the requirement that it be implemented as a codec miss the >>> point? >> >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. > > Yes - however, Victor currently apparently *doesn't* want it to be the > default, but wants the user to specify encoding="BOM". If so, it isn't > the default, and it is easy to implement as a codec. > >>> FWIW, I agree with Walter that if it is provided through the encoding= >>> argument, it should be a codec. If it is built into the open function >>> (for whatever reason), it must be provided by some other parameter. >> >> Why not simply encoding=None? > > I don't mind. Please re-read Walter's message - it only said that > *if* this is activated through encoding="BOM", *then* it must be > a codec, and could be on PyPI. I don't think Walter was talking about > the case "it is not activated through encoding='BOM'" *at all*. However if this autodetection feature is useful in other cases (no matter how it's activated), it should be a codec, because as part of the open() function it isn't reusable. >> The default value should provide the most useful >> behaviour possible. Forcing users to choose between two different autodetection >> strategies (encoding=None and another one) is a little insane IMO. And encoding="mbcs" is a third option on Windows. > That wouldn't disturb me much. There are a lot of things in that area > that are a little insane, starting with Microsoft Windows :-) ;) Servus, Walter From barry at python.org Mon Jan 11 13:37:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 07:37:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5DCE.3070909@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: On Jan 10, 2010, at 6:07 PM, Martin v. L?wis wrote: > As for decisions: I don't think there was an official BDFL pronouncent, > but I recall Guido posting a message close to that, proposing that 2.7 > will be a release that will see bug fix releases for an indefinite > period of time (where indefinite != infinite). This was shortly after > him proposing that perhaps we shouldn't make a 2.7 release at all, and > stop at 2.6. IMO, the focus for Python 2.7 (and beyond) must be on helping people transition to Python 3. That should be the reason why a 2.7 is released and, what should dictate whether a 2.8 is necessary. Maybe everything people need (except manpower and round tuits) is already there. I was pleasantly surprised to find that Distribute supports automatically running 2to3 at install time for Python packages. This allowed me to "port" a recent (admittedly small) package to Python 3 by making it "python2.6 -3" clean and adding a couple of attributes to my setup.py[1]. I don't know how widely that feature's known, but I think it's *huge*, and exactly what we need to be doing. The more we can make it easy for people to port their Python 2 code to Python 3, and to support that with one code base, the quicker we'll get to a predominantly Python 3 world. So the question we should be asking is: what's missing from Python 2.7 to help with this transition? If we can't get it into 2.7, then why, and would pushing it back to 2.8 help at all? -Barry [1] modulo a bug in Distribute that caused doctest in separate files to not be used when running under Python 3. From solipsis at pitrou.net Mon Jan 11 13:39:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 13:39:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <1263213574.3327.0.camel@localhost> > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. It is reusable as part of io.TextIOWrapper, though. Regards Antoine. From regebro at gmail.com Mon Jan 11 13:45:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 13:45:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. But an autodetect feature is not a codec. Sure it should be reusable, but making it a codec seems to be a weird hack to me. And how would you reuse it if it was a codec? A reusable autodetect feature would be useable to detect what codec it is. A autodetect codec would not be useful for that, as it would simply just decode. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 14:21:15 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 14:21:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B25CB.5030003@livinglogic.de> On 11.01.10 13:45, Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: >> However if this autodetection feature is useful in other cases (no >> matter how it's activated), it should be a codec, because as part of the >> open() function it isn't reusable. > > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. I think we already had this discussion two years ago in the context of XML decoding ;): http://mail.python.org/pipermail/python-dev/2007-November/075138.html > And how would > you reuse it if it was a codec? A reusable autodetect feature would be > useable to detect what codec it is. A autodetect codec would not be > useful for that, as it would simply just decode. I have implemented an XML codec (as part of XIST: http://pypi.python.org/pypi/ll-xist), that can do that: >>> from ll import xml_codec >>> import codecs >>> c = codecs.getincrementaldecoder("xml")() >>> c.encoding >>> c.decode(">> c.encoding >>> c.decode(" version='1.0'") u'' >>> c.encoding >>> c.decode(" encoding='iso-8859-1'?>") u"" >>> c.encoding 'iso-8859-1' Servus, Walter From regebro at gmail.com Mon Jan 11 14:47:12 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 14:47:12 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B25CB.5030003@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B25CB.5030003@livinglogic.de> Message-ID: <319e029f1001110547n1d1c9831p34b0eac519d13f9d@mail.gmail.com> On Mon, Jan 11, 2010 at 14:21, Walter D?rwald wrote: > I think we already had this discussion two years ago in the context of > XML decoding ;): Yup. Ans Martins answer then is my answer now: "> So the code is good, if it is inside an XML parser, and it's bad if it > is inside a codec? Exactly so. This functionality just *isn't* a codec - there is no encoding. Instead, it is an algorithm for *detecting* an encoding." The conclusion was that a method do autodetect encodings would be good. I think the same conclusion applies here. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 18:16:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:16:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B5CF5.50806@v.loewis.de> > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. Well, the existing UTF-16 codec also is an autodetect feature (to detect the endianness), and I don't consider it a weird hack. Regards, Martin From regebro at gmail.com Mon Jan 11 18:27:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:27:01 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B5CF5.50806@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> Message-ID: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> On Mon, Jan 11, 2010 at 18:16, "Martin v. L?wis" wrote: >> But an autodetect feature is not a codec. Sure it should be reusable, >> but making it a codec seems to be ?a weird hack to me. > > Well, the existing UTF-16 codec also is an autodetect feature (to > detect the endianness), and I don't consider it a weird hack. So the BOM codec should raise a UnicodeDecodeError if there is no BOM? Because that's what it would have to do, in that case, because it can't fall back on anything, it has to handle and implement all encodings that have a BOM. And is it then actually very useful? You would have to do a try/except first with encoding='BOM' and then encoding=None to get the fallback to the standard. I must say that I find this whole thing pretty obvious. 'BOM' is not an encoding. Either there should be a method to get the encoding from the BOM, returning None of there isn't one, or open() should look at the BOM when you pass in encoding=None. Or both. That covers all usecases, is easy and obvious. Either open(file=foo, encoding=None) or open(file, encoding=encoding_from_bom(file)) I can't see that open(file, encoding='BOM') has any benefit over this, covers any extra usecase and is clearer in any way. Instead it adds something confusing: An encoding that isn't an encoding. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From python at mrabarnett.plus.com Mon Jan 11 18:35:35 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:35:35 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <4B4B6167.40606@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: >> UTF-8 might be a good choice > > No, fallback if there is no BOM should be the local settings, just as > fallback is today if you don't specify a codec. > I mean, what if you want to look for a BOM but fall back to something > else? How far will we go with encoding special information in the > codecs names? codec='BOM else UTF-16 else locale'? :-) > > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. It should either be default to look for a > BOM when codec=None, or a special parameter. If none of these are > desired, then we need a special function that takes a filename or file > handle, and looks for a BOM and returns the codec found or None. But > I find that much less natural and obvious than checking the BOM when codec=None. > Or pass a function that accepts a byte stream or the first few bytes and returns the encoding and any unused bytes (because the byte stream might not be seekable)? def guess_encoding(byte_stream): data = byte_stream.read(2) if data == b"\xFE\xFF": return "UTF-16BE", b"" return "UTF-8", data text_file = open(filename, encoding=guess_encoding) ... From python at mrabarnett.plus.com Mon Jan 11 18:46:30 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:46:30 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> Message-ID: <4B4B63F6.1020309@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" > wrote: >> I know you are just looking for a compromise, but this shouldn't be >> it: the PSF has deliberately stayed out of the actual Python >> engineering, so the release that Benjamin makes is not done by the >> PSF (but by Benjamin and his contributors :-). > > Hm. Yeah. That's right of course. I started with saying "official", > but then I thought "official according to who?". :) "Official" is whatever the BDFL says it is! :-) > So I changed it to mentioning PSF, but that doesn't work either. I > guess the current writing as as good as it gets, unless we change > "scheduled" to "expected" or something. > From martin at v.loewis.de Mon Jan 11 18:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:59:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> Message-ID: <4B4B66EC.7000203@v.loewis.de> > I must say that I find this whole thing pretty obvious. 'BOM' is not > an encoding. That I certainly agree with. > That covers all usecases, is easy and obvious. Either open(file=foo, > encoding=None) or open(file, encoding=encoding_from_bom(file)) > > I can't see that open(file, encoding='BOM') has any benefit over this, Well, it would have the advantage that Walter pointed out: you can implement it independent of the open() implementation, and even provide it in older versions of Python. Regards, Martin From regebro at gmail.com Mon Jan 11 18:59:52 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B63F6.1020309@mrabarnett.plus.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> Message-ID: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> On Mon, Jan 11, 2010 at 18:46, MRAB wrote: > "Official" is whatever the BDFL says it is! :-) Heh, right. So, it should say "Guido wants 2.7 to be the last main version of Python 2, so it probably will be. We promise to release bugfixes it for, like, ages". No need to be needlessly formal. :-D -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From olemis at gmail.com Mon Jan 11 19:58:01 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 13:58:01 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> [...] > I had similar issues too (please read below ;o) ... On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > About guessing the encoding, I experienced this issue while I was developing a Trac plugin. What I was doing is as follows : - I guessed the MIME type + charset encoding using Trac MIME API (it was a CSV file encoded using UTF-16) - I read the file using `open` - Then wrapped the file using `codecs.EncodedFile` - Then used `csv.reader` ... and still get the BOM in the first value of the first row in the CSV file. {{{ #!python >>> mimetype 'utf-16-le' >>> ef = EncodedFile(f, 'utf-8', mimetype) }}} IMO I think I am +1 for leaving `open` just like it is, and use module `codecs` to deal with encodings, but I am strongly -1 for returning the BOM while using `EncodedFile` (mainly because encoding is explicitly supplied in ;o) > --Guido > CMIIW anyway ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mal at egenix.com Mon Jan 11 21:44:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Jan 2010 21:44:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> Message-ID: <4B4B8DB2.1060102@egenix.com> Olemis Lang wrote: >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>> BOM should be "ignored". >>> > [...] >> > > I had similar issues too (please read below ;o) ... > > On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> > > About guessing the encoding, I experienced this issue while I was > developing a Trac plugin. What I was doing is as follows : > > - I guessed the MIME type + charset encoding using Trac MIME API (it > was a CSV file encoded using UTF-16) > - I read the file using `open` > - Then wrapped the file using `codecs.EncodedFile` > - Then used `csv.reader` > > ... and still get the BOM in the first value of the first row in the CSV file. You didn't say, but I presume that the charset guessing logic returned either 'utf-16-le' or 'utf-16-be' - those encodings don't remove the leading BOM. The 'utf-16' codec will remove the BOM. > {{{ > #!python > >>>> mimetype > 'utf-16-le' >>>> ef = EncodedFile(f, 'utf-8', mimetype) > }}} Same here: the UTF-8 codec will not remove the BOM, you have to use the 'utf-8-sig' codec for that. > IMO I think I am +1 for leaving `open` just like it is, and use module > `codecs` to deal with encodings, but I am strongly -1 for returning > the BOM while using `EncodedFile` (mainly because encoding is > explicitly supplied in ;o) Note that EncodedFile() doesn't do any fancy BOM detection or filtering. This is the job of the codecs. Also note that BOM removal is only valid at the beginning of a file. All subsequent BOM-bytes have to be read as-is (they map to a zero-width non-breaking space) - without removing them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 11 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Jan 11 22:12:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 07:12:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <4B4B9447.4060101@gmail.com> Benjamin Peterson wrote: > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors I would call it a documentation bug: by leaving out the __get__ you're telling Python to override *just* the setting of the attribute, while still using the normal lookup process for retrieving the attribute (i.e. allowing the attribute to be shadowed in the instance dictionary). Adding a "print('Descriptor Invoked')" to the __set__ method in your example: >>> x = X() >>> print(x.attr) <__main__.Descr object at 0x7f283b51ac50> >>> x.attr = 42 Descriptor invoked >>> print(x.attr) 42 >>> x.attr = 6*9 Descriptor invoked >>> print(x.attr) 54 Note that the behaviour here is still different from that of a data descriptor: with a data descriptor, once it gets shadowed in the instance dictionary, the descriptor is ignored *completely*. The only way to get the descriptor involved again is to eliminate the shadowing. The non-data descriptor with only __set__ is just choosing not to override the attribute lookup process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Mon Jan 11 22:29:38 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 16:29:38 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B8DB2.1060102@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> <4B4B8DB2.1060102@egenix.com> Message-ID: <24ea26601001111329i7f609aa1i9f5db7eb61f1fae7@mail.gmail.com> Probably one part of this is OT , but I think it could complement the discussion ;o) On Mon, Jan 11, 2010 at 3:44 PM, M.-A. Lemburg wrote: > Olemis Lang wrote: >>> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >>> wrote: >>>> Hi, >>>> >>>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>>> BOM should be "ignored". >>>> >> [...] >>> >> >> I had similar issues too (please read below ;o) ... >> >> On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >>> >> >> About guessing the encoding, I experienced this issue while I was >> developing a Trac plugin. What I was doing is as follows : >> >> - I guessed the MIME type + charset encoding using Trac MIME API (it >> was a CSV file encoded using UTF-16) >> - I read the file using `open` >> - Then wrapped the file using `codecs.EncodedFile` >> - Then used `csv.reader` >> >> ... and still get the BOM in the first value of the first row in the CSV file. > > You didn't say, but I presume that the charset guessing logic > returned either 'utf-16-le' or 'utf-16-be' Yes. In fact they return the full mimetype 'text/csv; charset=utf-16-le' ... ;o) > - those encodings don't > remove the leading BOM. ... and they should ? > The 'utf-16' codec will remove the BOM. > In this particular case there's nothing I can do, I have to process whatever charset is detected in the input ;o) >> {{{ >> #!python >> >>>>> mimetype >> 'utf-16-le' >>>>> ef = EncodedFile(f, 'utf-8', mimetype) >> }}} > > Same here: the UTF-8 codec will not remove the BOM, you have > to use the 'utf-8-sig' codec for that. > >> IMO I think I am +1 for leaving `open` just like it is, and use module >> `codecs` to deal with encodings, but I am strongly -1 for returning >> the BOM while using `EncodedFile` (mainly because encoding is >> explicitly supplied in ;o) > > Note that EncodedFile() doesn't do any fancy BOM detection or > filtering. ... directly. > This is the job of the codecs. > OK ... to come back to the scope of the subject, in the general case, I think that BOM (if any) should be handled by codecs, and therefore indirectly by EncodedFile . If that's a explicit way of working with encodings I'd prefer to use that wrapper explicitly in order to (encode | decode) the file and let the codec detect whether there's a BOM or not and ?adjust? `tell`, `read` and everything else in that wrapper (instead of `open`). > Also note that BOM removal is only valid at the beginning of > a file. All subsequent BOM-bytes have to be read as-is (they > map to a zero-width non-breaking space) - without removing them. > ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Test cases for custom query (i.e report 9) ... PASS (1.0.0) - http://simelo.hg.sourceforge.net/hgweb/simelo/trac-gviz/rev/d276011e7297 From martin at v.loewis.de Mon Jan 11 22:42:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:42:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: <4B4B9B55.1010709@v.loewis.de> > So the question we should be asking is: what's missing from Python > 2.7 to help with this transition? Wrt. the distribute feature you've noticed: Python 3.0 already supports that in distutils. So from day one of Python 3.x, you could have used that approach for porting Python code. I had been promoting it ever since, but it's taking up only slowly, partially because it has competition in the approaches "burn the bridges" (i.e. convert to 3.x once, and then have two code bases, or abandon 2.x), and "avoid 2to3" (i.e. try to write code that works in both 2.x and 3.x unmodified). > If we can't get it into 2.7, then > why, and would pushing it back to 2.8 help at all? I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: a) telling people that they have to move to 2.6 first actually hurts migration, instead of helping, because it implies to them that they have to drop old versions (e.g. 2.3.) - just because they had *always* dropped old versions before supporting new ones. b) IMO, people also don't gain much by first migrating to 2.6. In principle, it gives them the opportunity to get py3k warnings. However, I haven't heard a single positive report where these warnings have actually helped people in porting. Yours is the first report saying that you followed the official guideline, but you didn't state whether doing so actually helped (or whether you just ported to 2.6 because the guideline told you to). c) whatever 2.7 does (except perhaps for the warnings), it won't be useful to applications, because they couldn't use it, anyway: they need to support 2.4 and 2.5, and it won't have any of the gimmicks people come up with for 2.7. Adding gimmicks to 2.7 might actually hurt porting: people may have to special-case 2.7 because their work-arounds for older versions may break in 2.7 (e.g. testing whether a name is *not* defined, when it becomes defined in 2.7), plus it gives them an incentive to not port yet until they can drop support for anything before 2.7. Inherently, 2.8 can't improve on that. Regards, Martin From martin at v.loewis.de Mon Jan 11 22:44:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:44:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> Message-ID: <4B4B9BB8.3070505@v.loewis.de> > So, it should say "Guido wants 2.7 to be the last main version of > Python 2, so it probably will be. We promise to release bugfixes it > for, like, ages". > > No need to be needlessly formal. :-D That summarizes my understanding of what Guido said fairly well :-) +1 for putting it into the announcements. Regards, Martin From fuzzyman at voidspace.org.uk Mon Jan 11 23:11:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 11 Jan 2010 22:11:44 +0000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4B9447.4060101@gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> Message-ID: <4B4BA220.20701@voidspace.org.uk> On 11/01/2010 21:12, Nick Coghlan wrote: > Benjamin Peterson wrote: > > My question is: Is this a doc bug or a implementation bug? If the > >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors >> > [snip...] > > Note that the behaviour here is still different from that of a data > descriptor: with a data descriptor, once it gets shadowed in the > instance dictionary, the descriptor is ignored *completely*. The only > way to get the descriptor involved again is to eliminate the shadowing. > The non-data descriptor with only __set__ is just choosing not to > override the attribute lookup process. > > Does that mean we need a third class of descriptors that are neither data descriptors nor non-data descriptors? What should we call them: really-not-data-descriptors? All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Mon Jan 11 23:55:01 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Jan 2010 14:55:01 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9BB8.3070505@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: +1 from me. (With the caveat that "time will tell" is definitely the ultimate answer. Plans may change unexpectedly, even though we don't expect them to.) PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not helpful. But I trust he has ported a lot more code to 3.x than I have personally. Are there other experiences? --Guido On Mon, Jan 11, 2010 at 1:44 PM, "Martin v. L?wis" wrote: >> So, it should say "Guido wants 2.7 to be the last main version of >> Python 2, so it probably will be. We promise to release bugfixes it >> for, like, ages". >> >> No need to be needlessly formal. :-D > > That summarizes my understanding of what Guido said fairly well :-) > > +1 for putting it into the announcements. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Jan 12 00:16:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 00:16:47 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <4B4BB15F.5020807@v.loewis.de> Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. That's really because I haven't even attempted to use it. Instead, the software would fall flat on its own when running the test suite in 3.x. IOW, I don't need 2.6 to tell me what might break when I can see 3.x actually breaking. Regards, Martin From david.lyon at preisshare.net Tue Jan 12 00:22:42 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:22:42 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Hi Martin, > Of course, the less active fraction of Python contributors may not > notice, since they just chose to not contribute (which, of course, > is fine). However, asking me to work twice as much as I want to > on the project to keep two branches alive is just unfair. Totally true. Actually as an end-developer I'd say that python 2.x series from a programming perspective is quite good. It doesn't need the addition of a lot of new features imho. So for me, I think too much time spent there would be not yield great benefits. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Well, we all know your work is super quality. :-) That's not being contested. However, Quality can be measured different ways and it can be assessed in different ways. Quality itself is a subjective thing. The point I'm only making is that if a piece of software doesn't have "new" things added over time, then users can get a reverse impression of a lack of quality. We've all seen where 'internal' quality can increase and user perceptions can decrease. It could be things like improved graphics and things readily apparent to the user. At the moment, I would say that the "internal" quality of the python 2.x series is super high. "external" quality issues such as the packaging dilemma give the user the opposite quality experience. But people are working on that as best they can elsewhere. I'll leave it at that. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Python 3.x needs more carrots. >From an ordinary (perphaps ignorant) user perspective there is nothing. Yes, we know if we actually will start programming then we will like it more. But my wishes to Santa Claus would be allow the free flow of PEPs for Python 3 packaging. Even encourage it. As an end developer, here's what I'd like from Santa in 2010 to get me to swap to python 3: * get all the packages on pypi tested for python 3 * put a web based package manager in python 3. This would perhaps be based around PIP (keep many people happy) and would look much like the built in web-console that you get with a $200 router. * Incorporate SCM based end-user software installs by adding to python3-stdlib the SCM support packages (cvs,bzr,svn,hg). This would *really* help in the scientific community. * put a web interface on distutils also so that we don't have to use a command line interface. (I want a pic of a smiley girl to great me when I build something - "Are you ready to build now Honey?"). ok - I joke. But the point is made. So, ok, maybe these things aren't about 'code' quality. But rather user experience. Things like these do count also as "quality" via the technical term "perception of quality". If the PEP process is as unblocked as the documentation implies, implying that anybody can contribute to Python 3. Then there shouldn't be any issue. David From solipsis at pitrou.net Tue Jan 12 00:37:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 23:37:47 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon preisshare.net> writes: > > > This has nothing to do with pushing 3.x, but all with managing > > available manpower and still providing quality software. > > Python 3.x needs more carrots. As someone who experiences the difference almost every day, I can say 3.x definitely has enough carrots for me. The simple fact that I will be able to raise exceptions with non-ASCII error messages without having to workaround a hopelessly broken conversion/reporting logic is a huge improvement. And that's only a small part of the picture. Regards Antoine. From janssen at parc.com Tue Jan 12 00:47:55 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 11 Jan 2010 15:47:55 PST Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <17215.1263253675@parc.com> > David Lyon preisshare.net> writes: > > > > > This has nothing to do with pushing 3.x, but all with managing > > > available manpower and still providing quality software. > > > > Python 3.x needs more carrots. I'd be happy to move UpLib to Python 3, when the various packages that I need are also on Python 3. And that depresses me to think about. I need Medusa ReportLab PyLucene Email Vobject Mutagen Pyglet Hachoir Win32 I'm on the mailing lists for a lot of these, and the only one that I know is thinking about Python 3 is Email. I'd expect Win32 and PIL to also be thinking about it, but I haven't heard anything. Bill From david.lyon at preisshare.net Tue Jan 12 00:47:51 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:47:51 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <1220.218.214.45.58.1263253671.squirrel@syd-srv02.ezyreg.com> Antoine writes: > As someone who experiences the difference almost every day, I can say 3.x > definitely has enough carrots for me. The simple fact that I will be able > to > raise exceptions with non-ASCII error messages without having to > workaround a > hopelessly broken conversion/reporting logic is a huge improvement. And > that's > only a small part of the picture. In no way could I disagree with you. Ascii works fine for us here - but I guess we're lucky. Just keep pushing python 3 and have a nice day.. David From barry at python.org Tue Jan 12 01:11:28 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:11:28 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100111191128.39020d89@freewill> On Jan 11, 2010, at 10:42 PM, Martin v. L?wis wrote: >Wrt. the distribute feature you've noticed: Python 3.0 already supports >that in distutils. So from day one of Python 3.x, you could have used >that approach for porting Python code. I had been promoting it ever >since, but it's taking up only slowly, partially because it has >competition in the approaches "burn the bridges" (i.e. convert to 3.x >once, and then have two code bases, or abandon 2.x), and "avoid 2to3" >(i.e. try to write code that works in both 2.x and 3.x unmodified). Interesting. The only reason I never used it before is because I didn't know about it. Am I alone? Maybe I'm projecting my own preferences, but it seems to me that supporting both Python 2 and 3 from the same code base would be a very powerful way to enable a large amount of existing code to support Python 3. I'm certainly going to do this from now on with all the libraries I maintain. I don't have any cycles to write code twice and I can't abandon Python 2 yet. I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. As an example, the one library I've already ported used a metaclass. I don't see any way to specify that the metaclass should be used in a portable way. In Python 2.6 it's: class Foo: __metaclass__ = Meta and in Python 3 it's: class Foo(metaclass=Meta): 2to3 made that pain go away. >I've done a fair bit of 3.x porting, and I'm firmly convinced that >2.x can do nothing: > >a) telling people that they have to move to 2.6 first actually > hurts migration, instead of helping, because it implies to them > that they have to drop old versions (e.g. 2.3.) - just because > they had *always* dropped old versions before supporting new ones. Is it just an implication, or is it reality? I have yet to try to support older versions of Python and also support Python 3. (The one package I've done this with so far only supports Python 2.6.) >b) IMO, people also don't gain much by first migrating to 2.6. > In principle, it gives them the opportunity to get py3k warnings. > However, I haven't heard a single positive report where these > warnings have actually helped people in porting. Yours is the > first report saying that you followed the official guideline, > but you didn't state whether doing so actually helped (or whether > you just ported to 2.6 because the guideline told you to). Python 2.6 has other useful features, which I want to take advantage of, so getting py3k warnings is a bonus. Running 'python2.6 -3 setup.py test' over my code did in fact help clean up a couple of problems. Seems like a good first step when considering Python 3 support to me. >c) whatever 2.7 does (except perhaps for the warnings), it won't > be useful to applications, because they couldn't use it, anyway: > they need to support 2.4 and 2.5, and it won't have any of the > gimmicks people come up with for 2.7. Adding gimmicks to 2.7 > might actually hurt porting: people may have to special-case > 2.7 because their work-arounds for older versions may break in 2.7 > (e.g. testing whether a name is *not* defined, when it becomes > defined in 2.7), plus it gives them an incentive to not port > yet until they can drop support for anything before 2.7. > >Inherently, 2.8 can't improve on that. I'm not so sure. Yes, as a package maintainer there are older versions to think about, but time moves on for everyone (hopefully :). By the time 2.8 is released, what will be the minimum version of Python provided by most OS vendors (where the majority of Python users probably get their 'python')? I guess some people will have to support everything from Python 2.3 to 2.8 but you're talking supporting something like a spread of 7 years of Python versions. What other platform do you support for 7 years? Even Ubuntu long term support (LTS) releases only live for 5 years and even then, newer Pythons are often back ported. Or if not, then who cares? You're not going to use a newer version of a library on an LTS anyway. I'm not necessarily saying that justifies a 2.8 release. I'm just asking how we can make it easier for people to port to Python 3. The automatic 2to3 has already made it easier for me to port one library to Python 3 because I barely had to even think about it. Maybe that's enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 01:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:12:19 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <20100111191219.5fdd2542@freewill> On Jan 11, 2010, at 02:55 PM, Guido van Rossum wrote: >PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not >helpful. But I trust he has ported a lot more code to 3.x than I have >personally. Are there other experiences? I've only done one small library so far, but it was helpful to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From andrew at bemusement.org Tue Jan 12 01:07:26 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Tue, 12 Jan 2010 11:07:26 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100112000726.GO32351@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > I've done a fair bit of 3.x porting, and I'm firmly convinced that > 2.x can do nothing: [...] > Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. From solipsis at pitrou.net Tue Jan 12 01:28:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 00:28:26 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: Andrew Bennetts bemusement.org> writes: > > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. I don't think a 2.8 could make you much closer to 3.x. AFAICT, every possible way to ease transition to 3.x will already be in 2.7, or almost. But perhaps I'm lacking imagination. Regards Antoine. From brian.curtin at gmail.com Tue Jan 12 02:57:46 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 11 Jan 2010 19:57:46 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > -Brett > I think it would be interesting to see how people are using the tracker, or how they want to be using it. For example, there are currently over 1500 open issues with no stage set, some of which seemingly haven't been read by anyone at all. Would a properly set stage field save issues from falling into a black hole? Food for thought: according to the last tracker summary, there are over 1000 open issues with a patch, and issues stay open an average of 700 days (median 450). Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Jan 12 04:53:24 2010 From: jackdied at gmail.com (Jack Diederich) Date: Mon, 11 Jan 2010 22:53:24 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. [sidebar] 1) the metaclass fixer was a PITA to implement. 2) 95% of __metaclass__ definitions searchable via google code were of the "__metaclass__ = type" variety. The 2to3 patch exists only because of the few other uses. 3) 100% of the module level assignments in public projects were the "__metaclass__ = type" variety which is why there isn't a fixer for that. Also, a fixer would have been really, really ugly (munge every class definition in this module because there is a top level assignment). -Jack From benjamin at python.org Tue Jan 12 05:01:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 11 Jan 2010 22:01:40 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <1afaf6161001112001t5e7bab83t7d93f33fa11ce849@mail.gmail.com> 2010/1/11 Jack Diederich : > On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: >> As an example, the one library I've already ported used a metaclass. ?I don't >> see any way to specify that the metaclass should be used in a portable way. >> In Python 2.6 it's: >> >> class Foo: >> ? ?__metaclass__ = Meta >> >> and in Python 3 it's: >> >> class Foo(metaclass=Meta): >> >> 2to3 made that pain go away. > > [sidebar] > 1) the metaclass fixer was a PITA to implement. Does this make it any less useful, though? :) -- Regards, Benjamin From steven.bethard at gmail.com Tue Jan 12 06:57:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 11 Jan 2010 21:57:18 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 4:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. Actually there's a solution to this one too: FooBase = Meta('FooBase', (), {}) class Foo(FooBase): ... That should work in Python 2.X and 3.X. I've got argparse running on Python 2.3-3.1, and the changes were pretty easy. You can see them all in the revision here: http://code.google.com/p/argparse/source/detail?r=12 I have aspirations of putting all of the tricks I learned up up on the Wiki somewhere, but I just haven't had the time. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From regebro at gmail.com Tue Jan 12 07:30:10 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:30:10 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> On Mon, Jan 11, 2010 at 23:55, Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. But I trust he has ported a lot more code to 3.x than I have > personally. Are there other experiences? It doesn't warn for that many of the unportable problems, but I'm not sure it can warn for them either. It's certainly helpful, just not very much. :) I think the biggest help was added in 2.6.2, and that's warning for old integer division. It will also warn for modules that aren't supported anymore, if I remember correctly, and that's often helpful. But it won't warn for real tricky problems, like binary vs text in strings, and I don't see how it can either. And, I just realized, it doesn't warn for you using cmp or __cmp__ either, and 2to3 won't fix that, so it should actually warn for it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:49:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:49:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> On Tue, Jan 12, 2010 at 01:11, Barry Warsaw wrote: > Interesting. ?The only reason I never used it before is because I didn't know > about it. ?Am I alone? Maybe not, but the Distribute feature is there because IMO the distutils feature by itself isn't particularily useful. You need to write your own distutils extensions, in practice, and they are not trivial. Distribute has simply done it for you. :) > I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:53:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:53:20 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <319e029f1001112253x511f8109ja2300a022010ef99@mail.gmail.com> On Tue, Jan 12, 2010 at 01:07, Andrew Bennetts wrote: > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. ?I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. When 2.7 was discussed several people agreed that 2.7 should include as much 3.x stuff as possible to ease transition. That turned out to not be very much, so I'm not sure there is more. :) Unless of course, 2.8 starts including more of the refactored libraries, but that's a very minor issue in porting, so it won't help much. To really help, it needs to start implement things that break backwards compatibility. That would be weird. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Tue Jan 12 10:39:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:39:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4BA220.20701@voidspace.org.uk> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> <4B4BA220.20701@voidspace.org.uk> Message-ID: <4B4C435B.7080903@gmail.com> Michael Foord wrote: >> Note that the behaviour here is still different from that of a data >> descriptor: with a data descriptor, once it gets shadowed in the >> instance dictionary, the descriptor is ignored *completely*. The only >> way to get the descriptor involved again is to eliminate the shadowing. >> The non-data descriptor with only __set__ is just choosing not to >> override the attribute lookup process. >> >> > > Does that mean we need a third class of descriptors that are neither > data descriptors nor non-data descriptors? Not really - leaving out __get__ when defining __set__ just creates a data descriptor that happens to use the default attribute look-up process rather than defining a different one. (Note that I had the data/non-data terminology backwards in my previous message - I tend to think of the two kinds of descriptor as enforced and non-enforced respectively precisely because I have to think about it to remember that "functions are non-data descriptors and properties are data descriptors, hence non-data descriptors only define __get__ while data descriptors define __set__". I don't find the data/non-data terminology intuitive at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:44:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:44:18 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> Message-ID: <4B4C4472.10104@gmail.com> Lennart Regebro wrote: > And, I just realized, it doesn't warn for you using cmp or __cmp__ > either, and 2to3 won't fix that, so it should actually warn for it. I have a vague recollection that we tried to warn for that and ended up nixing the warning due to vast swarms of false alarms (or because you couldn't write correct 2.x code without triggering it). I'd be happy for someone to prove my recollection wrong though (i.e. by writing a patch that implements such warnings). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:48:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:48:57 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4C4589.70906@gmail.com> David Lyon wrote: >> This has nothing to do with pushing 3.x, but all with managing >> available manpower and still providing quality software. > > Python 3.x needs more carrots. As Guido has said a few times, the gains are far greater for *new* Python developers than they are for existing ones. Existing developers have to unlearn old habits and wait for libraries they already use to get ported. New developers can just get started with a much cleaner language. They don't have as rich a 3rd party ecosystem on Python 3 as they would on Python 2.x at this point in time, but unlike existing developers they also have the luxury of cherry-picking just those packages that already have Python 3 support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Jan 12 11:20:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:20:27 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit?: > > For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. I think most issues /have/ been read. It's just that for many of them, nobody is interested enough in or feels competent enough for fixing them. > Would a properly set stage field save issues from > falling into a black hole? I don't think this has anything to do with properly setting the stage field. We just have limited time and manpower. Perhaps one of our goals should be to reach out more to potential contributors. > Food for thought: according to the last tracker summary, there are over > 1000 open issues with a patch, and issues stay open an average of 700 > days (median 450). As for open issues with a patch, a "code review" button sending this patch to Rietveld (or any other similar tool) is something many of us have been hoping for :-) It would make reviewing easier, faster and more thorough (because you visualize things better than by looking at a diff). Regards Antoine. From solipsis at pitrou.net Tue Jan 12 11:36:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:36:49 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Tue, 12 Jan 2010 10:20:27 +0000, Antoine Pitrou a ?crit?: > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Speaking of which, Steve had something to say about it: http://holdenweb.blogspot.com/2009/11/comments-or-not-public-or- private.html cheers Antoine. From ncoghlan at gmail.com Tue Jan 12 13:10:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 22:10:14 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C66A6.2040601@gmail.com> Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : >> For example, there are currently over >> 1500 open issues with no stage set, some of which seemingly haven't been >> read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. There are actually a whole host of reasons issues can stagnate: - a feature request may seem reasonable (hence it doesn't get rejected outright), but the right API may not be clear (hence it doesn't get implemented in the near term) - a patch may be reviewed and found to have significant defects (or simply be overreaching the stated goal) but the original patch poster loses interest after meeting resistance in their ambition to fix something that is "obviously" broken - a problem may simply be hard to fix in a backwards compatible way (or even at all!) Ultimately it boils down to Antoine's two categories (lack of interest and lack of relevant expertise to make a final decision) but there are a lot of different ways to get into those two buckets. Because we aren't ruthless about pruning those kinds of issues out of the tracker they're the ones that are going to accumulate over time. I'd actually be interested to know what the issue stats are like when RFEs are excluded and when the search is limited to the items flagged as 'easy'. If easy bug fix issues are taking a long time to get closed than that would bother me a lot more than RFEs that sit on the tracker for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 12 13:14:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:14:47 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071447.675c8f24@freewill> On Jan 11, 2010, at 10:53 PM, Jack Diederich wrote: >3) 100% of the module level assignments in public projects were the >"__metaclass__ = type" variety which is why there isn't a fixer for >that. Also, a fixer would have been really, really ugly (munge every >class definition in this module because there is a top level >assignment). And almost certainly unnecessary. IME, those are all to easily make bare class definitions new style in Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 13:16:09 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:16:09 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071609.1dcfffa6@freewill> On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: >Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > >That should work in Python 2.X and 3.X. Ugly, but good call! :) >I've got argparse running on Python 2.3-3.1, and the changes were >pretty easy. You can see them all in the revision here: > > http://code.google.com/p/argparse/source/detail?r=12 > >I have aspirations of putting all of the tricks I learned up up on the >Wiki somewhere, but I just haven't had the time. The more resources we can provide people, both in code and in documentation, the better. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Tue Jan 12 13:29:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 12:29:12 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112071609.1dcfffa6@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> Message-ID: <4B4C6B18.7050008@voidspace.org.uk> On 12/01/2010 12:16, Barry Warsaw wrote: > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > >> Actually there's a solution to this one too: >> >> FooBase = Meta('FooBase', (), {}) >> class Foo(FooBase): >> ... >> >> That should work in Python 2.X and 3.X. >> > Ugly, but good call! :) > > There are all sorts of tricks. For example you can do exception handling that works with pre-2.6 syntax and 3.0 with a bare except and using sys.exc_info. It is horrible, but acceptable for short pieces of code (I have a couple of small modules that do this). I haven't yet tried converting larger code-bases to Python 3, but I think the workflow advocated by Martin is greatly preferable to the hacks and tricks needed to make the same codebase run under 2 & 3. Michael >> I've got argparse running on Python 2.3-3.1, and the changes were >> pretty easy. You can see them all in the revision here: >> >> http://code.google.com/p/argparse/source/detail?r=12 >> >> I have aspirations of putting all of the tricks I learned up up on the >> Wiki somewhere, but I just haven't had the time. >> > The more resources we can provide people, both in code and in documentation, > the better. > > Thanks! > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 12 14:09:50 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Jan 2010 14:09:50 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C749E.4040009@egenix.com> Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > * argparse (PEP 389) > * brief mention on still wanting to break out the stdlib from CPython > * an official policy on extension modules? (i.e. must have a pure Python > implementation which can mean a ctypes implementation unless given an > explicit waiver) > > That's everything from a stdlib perspective. I have half-baked ideas, but > nothing concrete enough to bring up unless people really want to discuss > stuff like how to potentially standardize module deprecation warnings, etc. > > In terms of me personally, I do plan to bring up at some point during the > summit these points which don't squarely fit during my time slot: > > * an official unofficial policy on how new proposed features should come to > us (i.e. working code to python-ideas with a shell of a PEP that includes > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > If there is something missing from the stdlib discussion that you think > should be brought up at the summit let me know. And if there is something > here you want to discuss before the summit let me know and I can start a > separate thread on it. Could you please put these things and the results up on the Python wiki ?! We're going to have a language summit at EuroPython this year as well and may want to continue/extend the discussion based on what you're doing at PyCon. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 12 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brian.curtin at gmail.com Tue Jan 12 15:33:06 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 12 Jan 2010 08:33:06 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Tue, Jan 12, 2010 at 04:20, Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : > > > > For example, there are currently over > > 1500 open issues with no stage set, some of which seemingly haven't been > > read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. > > > Would a properly set stage field save issues from > > falling into a black hole? > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Agreed, I didn't mean to place blame on the stage field, I just ran with how I view that field since it was mentioned. When I'm thinking in "code-writing developer mode" (tm), I'm more likely to go to issues which have a stage so I know what I'm going into -- what needs to be worked on. When I'm in project cleanup mode, I go by stageless issues -- what is necessary for this to begin or end work. Maybe others work differently...software projects take all kinds :-) Anyways, sorry for the off-topic. If this is a summit worthy discussion, cool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 17:12:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 11:12:28 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C66A6.2040601@gmail.com> References: <4B4C66A6.2040601@gmail.com> Message-ID: <20100112161228.300EA1D688D@kimball.webabinitio.net> On Tue, 12 Jan 2010 22:10:14 +1000, Nick Coghlan wrote: > There are actually a whole host of reasons issues can stagnate: > - a feature request may seem reasonable (hence it doesn't get rejected > outright), but the right API may not be clear (hence it doesn't get > implemented in the near term) > - a patch may be reviewed and found to have significant defects (or > simply be overreaching the stated goal) but the original patch poster > loses interest after meeting resistance in their ambition to fix > something that is "obviously" broken > - a problem may simply be hard to fix in a backwards compatible way (or > even at all!) I would add: - a patch is reviewed but the reviewer requests a second opinion before committing, which never arrives, and the original reviewer forgets about the issue. I have occasionally observed apparently moribund issues spring to life and get resolved when a triage person does something as simple as updating the releases to which an issue applies. > Ultimately it boils down to Antoine's two categories (lack of interest > and lack of relevant expertise to make a final decision) but there are a > lot of different ways to get into those two buckets. There's a third bucket here: lack of time. Sometimes there is interest and expertise, but no time. Worse, sometimes we end up waiting on the person with the expertise and interest but no time, when someone else with somewhat less expertise but more time should just go ahead and do the commit. (*That* one should be fixable.) > Because we aren't ruthless about pruning those kinds of issues out of > the tracker they're the ones that are going to accumulate over time. Another other category that hangs around in the tracker is items for which there simply isn't a consensus. I suppose those could be brought to python-dev for a final decision if someone wanted to tackle that task. And I don't think it is a lack of ruthlessness. I think it is the current community consensus that we leave these issues open in the tracker in case someone does come along and want to deal with them. (*) > I'd actually be interested to know what the issue stats are like when > RFEs are excluded and when the search is limited to the items flagged as > 'easy'. If easy bug fix issues are taking a long time to get closed than > that would bother me a lot more than RFEs that sit on the tracker for > years. I'd also be interested to know what the average and median lifetime of closed bug reports is. Not the metrics for open issues, but the metrics for closed ones. My theory is that we close a lot of bugs fairly promptly, but the ones in the above categories make the average age of *open* issues high. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls (*) For example, I'm currently going through all the open issues relating to the email package, and some of them are pretty darn old, yet still have useful content. From brett at python.org Tue Jan 12 18:40:13 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:40:13 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C749E.4040009@egenix.com> References: <4B4C749E.4040009@egenix.com> Message-ID: On Tue, Jan 12, 2010 at 05:09, M.-A. Lemburg wrote: > Brett Cannon wrote: > > Michael has given me the hg transition/stdlib time slot at the language > > summit this year. In regards to that I plan to lead a discussion on: > > > > * where we are at w/ the Hg transition (Dirkjan should be there and I did > a > > blog post on this topic recently: > > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > > * argparse (PEP 389) > > * brief mention on still wanting to break out the stdlib from CPython > > * an official policy on extension modules? (i.e. must have a pure Python > > implementation which can mean a ctypes implementation unless given an > > explicit waiver) > > > > That's everything from a stdlib perspective. I have half-baked ideas, but > > nothing concrete enough to bring up unless people really want to discuss > > stuff like how to potentially standardize module deprecation warnings, > etc. > > > > In terms of me personally, I do plan to bring up at some point during the > > summit these points which don't squarely fit during my time slot: > > > > * an official unofficial policy on how new proposed features should come > to > > us (i.e. working code to python-ideas with a shell of a PEP that includes > > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > > * any changes needed to the issue tracker to help with the workflow? > (stage > > field seems like a failed experiment and we now have several effective > > triage people who can help w/ guiding changes) > > > > If there is something missing from the stdlib discussion that you think > > should be brought up at the summit let me know. And if there is something > > here you want to discuss before the summit let me know and I can start a > > separate thread on it. > > Could you please put these things and the results up on the Python > wiki ?! > > We're going to have a language summit at EuroPython this year > as well and may want to continue/extend the discussion based > on what you're doing at PyCon. > > I expect there will be at least summary emails on what gets discussed. There is also a chance that it will be videotaped. -Brett > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 12 2010) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 18:47:50 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:47:50 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? >> (stage field seems like a failed experiment and we now have several >> effective triage people who can help w/ guiding changes) >> >> -Brett >> > I think it would be interesting to see how people are using the tracker, or > how they want to be using it. For example, there are currently over 1500 > open issues with no stage set, some of which seemingly haven't been read by > anyone at all. Would a properly set stage field save issues from falling > into a black hole? > > "Using" is the key word there. I know this thread went on about how bugs tend to end up being left open, but what I was proposing to talk about was whether there is any shift desired by the seasoned tracker users to help in their work. For instance, the Stage field was meant to help, but I don't think it is really getting used much, which makes it at best just a UI junk and at worst something to confuse new users. So I just wanted to discuss things as a group to see if we could streamline some things, add others, etc. At worst I will try to grab people like Mark D., R. David, Antoine, Georg, etc. at the summit (not sure exactly which the heavy bug closers will be there), take them aside, and flat-out ask what they need to make their jobs easier. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 19:29:06 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:29:06 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C6B18.7050008@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> <4B4C6B18.7050008@voidspace.org.uk> Message-ID: On Tue, Jan 12, 2010 at 04:29, Michael Foord wrote: > On 12/01/2010 12:16, Barry Warsaw wrote: > > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > > > Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > > That should work in Python 2.X and 3.X. > > > Ugly, but good call! :) > > > > > There are all sorts of tricks. For example you can do exception handling > that works with pre-2.6 syntax and 3.0 with a bare except and using > sys.exc_info. It is horrible, but acceptable for short pieces of code (I > have a couple of small modules that do this). > > I haven't yet tried converting larger code-bases to Python 3, but I think > the workflow advocated by Martin is greatly preferable to the hacks and > tricks needed to make the same codebase run under 2 & 3. > > In other words we need to pull together a HOWTO for Python source like the one for extension modules that Benjamin wrote and make it rather prominently linked from the Python 3 documentation index page. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 19:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 13:31:23 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? > >> (stage field seems like a failed experiment and we now have several > >> effective triage people who can help w/ guiding changes) > >> > > I think it would be interesting to see how people are using the tracker, or > > how they want to be using it. For example, there are currently over 1500 > > open issues with no stage set, some of which seemingly haven't been read by > > anyone at all. Would a properly set stage field save issues from falling > > into a black hole? > > > "Using" is the key word there. I know this thread went on about how bugs > tend to end up being left open, but what I was proposing to talk about was > whether there is any shift desired by the seasoned tracker users to help in > their work. For instance, the Stage field was meant to help, but I don't > think it is really getting used much, which makes it at best just a UI junk > and at worst something to confuse new users. So I just wanted to discuss > things as a group to see if we could streamline some things, add others, > etc. Well, I for one find the stage field useful. It reminds me at a glance where the issue is at in the workflow (and 'not set' is a valid place in the workflow that an issue sometimes stays in for a while for reason other than not having been triaged yet). I suspect that if we discussed it as a group (we who make the most changes on the tracker) we could improve it, and the interface in general. By the way, you could talk to people who aren't going to be at the summit on #python-dev; I think all the currently tracker-active people hang out there on a regular basis. I'll have to give some thought to what changes/improvements might be most useful, now that I've been doing this for a while. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Tue Jan 12 19:34:02 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:34:02 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> References: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> Message-ID: On Tue, Jan 12, 2010 at 10:31, R. David Murray wrote: > On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > > On Mon, Jan 11, 2010 at 17:57, Brian Curtin > wrote: > > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > > >> * any changes needed to the issue tracker to help with the workflow? > > >> (stage field seems like a failed experiment and we now have several > > >> effective triage people who can help w/ guiding changes) > > >> > > > I think it would be interesting to see how people are using the > tracker, or > > > how they want to be using it. For example, there are currently over > 1500 > > > open issues with no stage set, some of which seemingly haven't been > read by > > > anyone at all. Would a properly set stage field save issues from > falling > > > into a black hole? > > > > > "Using" is the key word there. I know this thread went on about how bugs > > tend to end up being left open, but what I was proposing to talk about > was > > whether there is any shift desired by the seasoned tracker users to help > in > > their work. For instance, the Stage field was meant to help, but I don't > > think it is really getting used much, which makes it at best just a UI > junk > > and at worst something to confuse new users. So I just wanted to discuss > > things as a group to see if we could streamline some things, add others, > > etc. > > Well, I for one find the stage field useful. It reminds me at a glance > where the issue is at in the workflow (and 'not set' is a valid place > in the workflow that an issue sometimes stays in for a while for reason > other than not having been triaged yet). I suspect that if we discussed > it as a group (we who make the most changes on the tracker) we could > improve it, and the interface in general. > > By the way, you could talk to people who aren't going to be at the > summit on #python-dev; I think all the currently tracker-active people > hang out there on a regular basis. > > Sounds reasonable. > I'll have to give some thought to what changes/improvements might be > most useful, now that I've been doing this for a while. How about everyone who is active on bugs.python.org give it some thought and we can try to have a discussion at some point in the relatively near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jan 12 21:58:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 06:58:49 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: <4B4C749E.4040009@egenix.com> Message-ID: <4B4CE289.6040709@gmail.com> Brett Cannon wrote: > I expect there will be at least summary emails on what gets discussed. > There is also a chance that it will be videotaped. The Wiki makes for a better summary archive though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jan 12 22:53:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:53:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <4B4CEF3D.8000107@v.loewis.de> >> a) telling people that they have to move to 2.6 first actually >> hurts migration, instead of helping, because it implies to them >> that they have to drop old versions (e.g. 2.3.) - just because >> they had *always* dropped old versions before supporting new ones. > > Is it just an implication, or is it reality? That's only the implication. However, this was precisely the dialogue when talking to Django. If you start with "start supporting 2.6", the immediate response, without listening further, was, "ok, wait until we drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). Then explain it to the individual you are talking to, wait for the next developer of the project step along, and see how he brings up the very same line of thinking (supporting new versions == dropping support for old versions). I think only part of that comes from the maintenance burden. The other part is that they *want* to drop support for old versions, so that they can eventually start using new features (e.g. generator expressions). So they welcome the requirement to support new versions as an excuse to drop old ones ("it is obvious that you have to drop 2.3 to support 3.2"). However, their users then won't let them drop old versions. > >> b) IMO, people also don't gain much by first migrating to 2.6. >> In principle, it gives them the opportunity to get py3k warnings. >> However, I haven't heard a single positive report where these >> warnings have actually helped people in porting. Yours is the >> first report saying that you followed the official guideline, >> but you didn't state whether doing so actually helped (or whether >> you just ported to 2.6 because the guideline told you to). > > Python 2.6 has other useful features, which I want to take advantage of I think you are a minority with that, being able to actually use the 2.6 features already. Many projects can't, as they have to support at least 2.4 still (so the with statement is right out). >> Inherently, 2.8 can't improve on that. > > I'm not so sure. Yes, as a package maintainer there are older versions to > think about, but time moves on for everyone (hopefully :). By the time 2.8 is > released, what will be the minimum version of Python provided by most OS > vendors (where the majority of Python users probably get their 'python')? "Current" Linux distributions will have 2.6 then. "Old" installations will have 2.4. > I > guess some people will have to support everything from Python 2.3 to 2.8 but > you're talking supporting something like a spread of 7 years of Python > versions. What other platform do you support for 7 years? I think 2.3 will really be gone by the time 2.8 might get released. Even with 2.7, you'd end up with a span of seven years, though. Python had been supporting Windows 95 for more than 7 years (I think rather 9 or 10), likewise Windows 3.1 before that. Python 2.7 will likely still support Windows 2000, which then will be ten years old. Solaris support will probably go back to Solaris 2.6, which will be 13 years old when Python 2.7 gets released. It's only the Linux (and OS X) releases that move so quickly. Regards, Martin From martin at v.loewis.de Tue Jan 12 22:56:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:56:55 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> Message-ID: <4B4CF027.4080007@v.loewis.de> > Maybe not, but the Distribute feature is there because IMO the > distutils feature by itself isn't particularily useful. You need to > write your own distutils extensions, in practice, and they are not > trivial. I wouldn't say that. My Django port works with bare distutils (as does Django itself), and it works fine. That distribute had to redo it is only because setuptools *replaces* the build_py command, as does the 2to3 support in distutils. So only if you have a different build_py already, you can't use what is in distutils. Regards, Martin From fuzzyman at voidspace.org.uk Tue Jan 12 23:02:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:02:34 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CEF3D.8000107@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> Message-ID: <4B4CF17A.1000503@voidspace.org.uk> On 12/01/2010 21:53, "Martin v. L?wis" wrote: >>> a) telling people that they have to move to 2.6 first actually >>> hurts migration, instead of helping, because it implies to them >>> that they have to drop old versions (e.g. 2.3.) - just because >>> they had *always* dropped old versions before supporting new ones. >>> >> Is it just an implication, or is it reality? >> > That's only the implication. However, this was precisely the dialogue > when talking to Django. If you start with "start supporting 2.6", the > immediate response, without listening further, was, "ok, wait until we > drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). > > Then explain it to the individual you are talking to, wait for the next > developer of the project step along, and see how he brings up the > very same line of thinking (supporting new versions == dropping support > for old versions). > > I think only part of that comes from the maintenance burden. The other > part is that they *want* to drop support for old versions, so that they > can eventually start using new features (e.g. generator expressions). > So they welcome the requirement to support new versions as an excuse > to drop old ones ("it is obvious that you have to drop 2.3 to support > 3.2"). However, their users then won't let them drop old versions. > > > I agree with Martin that the *perception* is that to use Python 2.6 to help you port to Python 3 you have to be willing to drop support for earlier versions of Python. >> >>> b) IMO, people also don't gain much by first migrating to 2.6. >>> In principle, it gives them the opportunity to get py3k warnings. >>> However, I haven't heard a single positive report where these >>> warnings have actually helped people in porting. Yours is the >>> first report saying that you followed the official guideline, >>> but you didn't state whether doing so actually helped (or whether >>> you just ported to 2.6 because the guideline told you to). >>> >> Python 2.6 has other useful features, which I want to take advantage of >> > I think you are a minority with that, being able to actually use the 2.6 > features already. Many projects can't, as they have to support at least > 2.4 still (so the with statement is right out). > > Well, the IronPython community has almost completely moved over to IronPython 2.6. :-) We tend to ship our Python runtime with our applications though. As it happens I'm now working with CPython on the server (Python 2.5) and IronPython in the browser where we are using 2.6. The new property feature is nice, as is having with without a __future__ import. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From regebro at gmail.com Tue Jan 12 23:03:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 23:03:41 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF027.4080007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> <4B4CF027.4080007@v.loewis.de> Message-ID: <319e029f1001121403x14ef7ec8x729fb80fa67feaef@mail.gmail.com> On Tue, Jan 12, 2010 at 22:56, "Martin v. L?wis" wrote: >> Maybe not, but the Distribute feature is there because IMO the >> distutils feature by itself isn't particularily useful. You need to >> write your own distutils extensions, in practice, and they are not >> trivial. > > I wouldn't say that. My Django port works with bare distutils (as > does Django itself), and it works fine. > > That distribute had to redo it is only because setuptools *replaces* > the build_py command, as does the 2to3 support in distutils. So only > if you have a different build_py already, you can't use what is in > distutils. Yeah, you are right, I misremembered. The actual additional feature is the support for the test command. Testing under Python 2 and 3 without it is annoying. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 12 23:04:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:04:37 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <4B4CF1F5.4050403@v.loewis.de> > [...] >> I've done a fair bit of 3.x porting, and I'm firmly convinced that >> 2.x can do nothing: > [...] >> Inherently, 2.8 can't improve on that. > > I agree that there are limitations like the ones you've listed, but I > disagree with your conclusion. Maybe you assume that it's just as hard > to move to 2.8 (using the py3k backports not available in say 2.5) as it > is to 3.x? Not at all, no. I'd rather expect that code that runs on 2.7 will run on 2.8 unmodified. > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? > I think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? > Then, when all of your dependencies (or viable alternatives to those > dependencies) are available for 3.x, you'll have an easier transition if > you can start from a 2.x with fewer differences in features. But you won't *have* fewer differences. Just because your code runs on 2.8 doesn't mean it will stop running on 2.3 (if you have a need for that). This doesn't get you any closer - you can't use any of the 2.8 features as long as you have to support older versions of Python. > Fundamentally the more 2.x can converge on 3.x, the easier it will be > for users to make the leap, because it will be a smaller leap. No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. > The > longer the 2.x series lives, the more these newer 2.x versions like 2.7 > and maybe even 2.8 will be available on common platforms for people to > depend upon as minimum versions, which means that as time goes by they > can depend on a version that's closer to 3.x. No, that's incorrect. Suppose 2.7 is the last 2.x release, to be released in 2010. Then, in 2015, it may be that everybody has migrated to 2.7, which then is a common platform. If you release 2.8 in 2012, then, in 2015, people will be split between 2.7 and 2.8, and so there won't be a common platform before 2017. So stopping 2.x development *earlier* will also give us a common platform earlier. Regareds, Martin From martin at v.loewis.de Tue Jan 12 23:07:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:07:02 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4CF286.5040002@v.loewis.de> > I think it would be interesting to see how people are using the tracker, > or how they want to be using it. For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. Would a properly set stage field save issues from > falling into a black hole? I personally don't think this would be the case. What would help is people who actually *work* on the issues, rather than merely reading them. However, this being open source, there is no way to force it (unless you create an incentive, such as the five-for-one deal). Regards, Martin From python at mrabarnett.plus.com Tue Jan 12 23:10:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jan 2010 22:10:28 +0000 Subject: [Python-Dev] regex module Message-ID: <4B4CF354.2050603@mrabarnett.plus.com> Hi all, I'm back on the regex module after doing other things and I'd like your opinion on a number of matters: Firstly, the current re module has a bug whereby it doesn't split on zero-width matches. The BDFL has said that this behaviour should be retained by default in case any existing software depends on it. My question is: should my regex module still do this for Python 3? Speaking personally, I'd like it to behave correctly, and Python 3 is the version where backwards-compatibility is allowed to be broken. Secondly, Python 2 is reaching the end of the line and Python 3 is the future. Should I still release a version that works with Python 2? I'm thinking that it could be confusing if new regex module did zero-width splits correctly in Python 3 but not in Python 2. And also, should I release it only for Python 3 as a 'carrot'? Finally, the module allows some extra backslash escapes, eg \g, in the pattern. Should it treat ill-formed escapes, eg \g, as it would have treated them in the re module? Thanks From michael at voidspace.org.uk Tue Jan 12 23:46:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:46:49 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 Message-ID: <4B4CFBD9.1090009@voidspace.org.uk> I presume the email below is about the Windows binary. Does the AMD64 release work on intel 64bit and can we make the wording clearer on the download page? The current description is " Windows AMD64 binary". All the best, Michael -------- Original Message -------- Subject: Download Page - AMD64 Date: Fri, 11 Dec 2009 04:03:16 -0600 From: Thomas Brownback To: webmaster at python.org Should I use the AMD64 version of Python on an Intel 64 chip? I know those 64-bit implementations are very similar, but are they similar enough that your AMD64 will work on Intel? If so, perhaps a note should be placed on that page to help avoid confusion. Explicit being better than implicit and all. Thanks for your time, Thomas -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at bemusement.org Tue Jan 12 23:49:56 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Wed, 13 Jan 2010 09:49:56 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100112224956.GC28576@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > But a hypothetical 2.8 would also give people a way to move closer to > > py3k without giving up on using all their 2.x-only dependencies. > > How so? If they use anything that is new in 2.8, they *will* need to > drop support for anything before it, no??? > > > I think it's much more likely that libraries like Twisted can support 2.8 > > in the near future than 3.x. > > Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does > that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. From tjreedy at udel.edu Tue Jan 12 23:51:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 17:51:31 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: On 1/12/2010 5:04 PM, "Martin v. L?wis" wrote: > But you won't *have* fewer differences. Just because your code runs > on 2.8 doesn't mean it will stop running on 2.3 (if you have a need > for that). This doesn't get you any closer - you can't use any of > the 2.8 features as long as you have to support older versions of > Python. > >> Fundamentally the more 2.x can converge on 3.x, the easier it will be >> for users to make the leap, because it will be a smaller leap. > > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. > >> The >> longer the 2.x series lives, the more these newer 2.x versions like 2.7 >> and maybe even 2.8 will be available on common platforms for people to >> depend upon as minimum versions, which means that as time goes by they >> can depend on a version that's closer to 3.x. > > No, that's incorrect. Suppose 2.7 is the last 2.x release, to be > released in 2010. Then, in 2015, it may be that everybody has migrated > to 2.7, which then is a common platform. > > If you release 2.8 in 2012, then, in 2015, people will be split between > 2.7 and 2.8, and so there won't be a common platform before 2017. Just like people today may need to work with both 2.5 and 2.6, or privately backport 2.6 bugfixes to 2.5. > So stopping 2.x development *earlier* will also give us a common > platform earlier. With years of bug fixes and hence high quality. From tjreedy at udel.edu Wed Jan 13 00:05:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 18:05:12 -0500 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On 1/12/2010 5:10 PM, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. Are you writing a new module with a new name? If so, do you expect it to replace or augment re? (This is the same question as for optparse vs. argparse, which I understand to not yet be decided.) > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, 2.7 stdlib is already out. A new engine should get some community testing before going in the stdlib. Even 3.2 beta is not that far off (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? What does re do with analogous cases? Terry Jan Reedy From david.lyon at preisshare.net Wed Jan 13 00:20:54 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 13 Jan 2010 10:20:54 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C4589.70906@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> <4B4C4589.70906@gmail.com> Message-ID: <1333.218.214.45.58.1263338454.squirrel@syd-srv02.ezyreg.com> > Nick wrote: >>> This has nothing to do with pushing 3.x, but all with managing >>> available manpower and still providing quality software. >> >> Python 3.x needs more carrots. > > As Guido has said a few times, the gains are far greater for *new* > Python developers than they are for existing ones. Well both you and Guido are most likely 100% correct. :-) > They don't have as rich a 3rd party ecosystem > on Python 3 as they would on Python 2.x at this point in time, but > unlike existing developers they also have the luxury of cherry-picking > just those packages that already have Python 3 support. Most likely. I wouldn't want to say anything to discourage people from going to python 3. In other languages, I have much experience of making the jump to 'a whole new world'. It's unsettling at first, but after that, you suddenly find the new model has a better turbo than the last and you're at the next corner faster than you can think. So it's all good. But I still maintain Python 3.0 needs more carrots. For example, if mercurial or any other cool lib gets added to python 3 (and I can name a few that should go in python 3) then they should be added to python 3 and not to python 2.x They would serve as good carrots. Make fresh the python 3 stdlib and preserve the python 2.x stdlib. I really think we are somewhat short on resources to do what Guido has asked about bringing python up to CPAN level with respect to packages. We're starting a long way back with horses and swords and trying to catch a well fed and greased modern machine.. I'm sure we can get a modern package testbot operational for python. But I wish those who were complaining about the packaging problem so much could throw some money at the problem instead of moaning about it. As is done on other open-source projects. Some organisation would be beneficial. Finding funds so that a small group of people could work on the problem would be a great boost to forward progress. I just think python packaging needs six months of that to repair the years and years of neglect and stagnation. Even if it is only beer and bus fare money. It just needs a temporary shot of adrenalin in the arm.. so to speak.. Regards David From martin at v.loewis.de Wed Jan 13 00:28:14 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 00:28:14 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D058E.4030404@v.loewis.de> > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? "intel 64bit" is as clear as mud. It could mean the "Intel 64" architecture, or it could mean the "IA-64" architecture, both are 64-bit architectures with processors manufactured by Intel. The former is officially called AMD64 (not by Intel, but officially), and our AMD64 binaries run on such processors. The latter is also called "Itanium", and we stopped producing binaries for that architecture a while ago; the AMD64 binaries will *not* run on it. The wording could probably be changed to match the reader's expectation more, most likely, it would also get more incorrect in the process. To make it correct and explicit, an entire paragraph educating users about 64-bit architectures and that there are many of them and that they are mutually incompatible might be required. Most likely, Thomas' processor implements the AMD64 architecture, even though it is manufactured by Intel. A short sentence that he would probably understand (given that he used "Intel 64", not "64bit") is: """The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).""" Regards, Martin From martin at v.loewis.de Wed Jan 13 00:30:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 00:30:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112224956.GC28576@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> <20100112224956.GC28576@steerpike.home.puzzling.org> Message-ID: <4B4D0613.1010503@v.loewis.de> > I'm not talking about Twisted moving to 3.x (FWIW, I think the only > movement there so far is some patches for some -3 warnings). The > situation I'm describing is a project X that: > > (a) has 2.x-only dependencies, and > (b) would like to be as close as possible to 3.x (because they like > the new features and/or want to be as ready as possible to jump > when (a) is fixed). This assumes that jumping to 3.x is easier if you are closer to it. Please trust me that this is not the case. >> No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they >> likely won't. > > But this is my point. I think they would as an intermediate step to > jumping to 3.x (which also requires dropping 2.5, after all!), if for > some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. No, and no. No: it's not an intermediate step, and no, supporting 3.x does *not* require dropping 2.5. Regards, Martin From fuzzyman at voidspace.org.uk Wed Jan 13 00:40:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:40:35 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D058E.4030404@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> Message-ID: <4B4D0873.5070006@voidspace.org.uk> On 12/01/2010 23:28, "Martin v. L?wis" wrote: > [snip...] > """The binaries for AMD64 will also work on processors that implement > the Intel 64 architecture (formerly EM64T), i.e. the architecture that > Microsoft calls x64, and AMD called x86-64 before calling it AMD64. > They will not work on Intel Itanium Processors (formerly IA-64).""" > > To those not intimately aware of the history and present of 64 bit architecture this sentence would be a great addition - thanks. I'm adding it now as a footnote. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From lists at cheimes.de Wed Jan 13 00:41:04 2010 From: lists at cheimes.de (Christian Heimes) Date: Wed, 13 Jan 2010 00:41:04 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D0890.2030505@cheimes.de> Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". The installer works on all AMD64 compatible Intel CPUs. *scnr* As you most likely know all modern Intel 64bit CPUs are based on AMD's x86-64 design. Only the Itanium family is based on the other Intel 64bit design: IA-64. The name AMD64 was chosen because most Linux and BSD distributions call it so. The name 'AMD64' has caused confusion in the past because users thought that the installer works only on AMD CPUs. How about: * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / X86-64 binary -- does not include source) instead of: * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does not include source) ? Christia From fuzzyman at voidspace.org.uk Wed Jan 13 00:55:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:55:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0873.5070006@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> <4B4D0873.5070006@voidspace.org.uk> Message-ID: <4B4D0BD4.5050109@voidspace.org.uk> On 12/01/2010 23:40, Michael Foord wrote: > On 12/01/2010 23:28, "Martin v. L?wis" wrote: >> [snip...] >> """The binaries for AMD64 will also work on processors that implement >> the Intel 64 architecture (formerly EM64T), i.e. the architecture that >> Microsoft calls x64, and AMD called x86-64 before calling it AMD64. >> They will not work on Intel Itanium Processors (formerly IA-64).""" >> > > To those not intimately aware of the history and present of 64 bit > architecture this sentence would be a great addition - thanks. I'm > adding it now as a footnote. > I can add it now to the main download page and existing release pages - but are there changes needed to any scripts to change pages created for *new* releases? All the best, Michael Foord > All the best, > > Michael Foord > >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From python at mrabarnett.plus.com Wed Jan 13 01:22:01 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 00:22:01 +0000 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D1229.70708@mrabarnett.plus.com> Terry Reedy wrote: > On 1/12/2010 5:10 PM, MRAB wrote: >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. > > Are you writing a new module with a new name? If so, do you expect it to > replace or augment re? (This is the same question as for optparse vs. > argparse, which I understand to not yet be decided.) > It's a module called 'regex'. It can be used in place of 're' by using "import regex as re", except for differences such as "\g" being a legal group reference in pattern strings. >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? > > 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, > 2.7 stdlib is already out. A new engine should get some community > testing before going in the stdlib. Even 3.2 beta is not that far off > (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? > > What does re do with analogous cases? > The 're' module treats r"\g" as "g"; both 're' and 'regex' treat, say, r"\q" as "q". The closest analogue to what I'm asking about is that re treats the ill-formed repeat r"x{1," as a literal, which sort of suggests that r"\g" should be treated as "g", but r"\g" is now a group reference (re would treat that as "g". Does that sound reasonable? From fuzzyman at voidspace.org.uk Wed Jan 13 01:50:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 00:50:39 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D18DF.6070502@voidspace.org.uk> On 12/01/2010 23:41, Christian Heimes wrote: > Michael Foord wrote: > >> I presume the email below is about the Windows binary. Does the AMD64 >> release work on intel 64bit and can we make the wording clearer on the >> download page? >> >> The current description is " Windows AMD64 binary". >> > The installer works on all AMD64 compatible Intel CPUs. *scnr* > > As you most likely know all modern Intel 64bit CPUs are based on AMD's > x86-64 design. Only the Itanium family is based on the other Intel 64bit > design: IA-64. The name AMD64 was chosen because most Linux and BSD > distributions call it so. The name 'AMD64' has caused confusion in the > past because users thought that the installer works only on AMD CPUs. > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) > Right - I've made that change for the Python 2.6, 2.7, 3.0 and 3.1 download pages with a footnote. Prior to that we were offering ia64 *and* amd64 releases so I've left those unchanged. Thanks, Michael > ? > > Christia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From exarkun at twistedmatrix.com Wed Jan 13 02:40:03 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 13 Jan 2010 01:40:03 -0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100113014003.1898.1305834328.divmod.xquotient.219@localhost.localdomain> On 12 Jan, 10:04 pm, martin at v.loewis.de wrote: >>[...] >>>I've done a fair bit of 3.x porting, and I'm firmly convinced that >>>2.x can do nothing: >>[...] >>>Inherently, 2.8 can't improve on that. >> >>I agree that there are limitations like the ones you've listed, but I >>disagree with your conclusion. Maybe you assume that it's just as >>hard >>to move to 2.8 (using the py3k backports not available in say 2.5) as >>it >>is to 3.x? > >Not at all, no. I'd rather expect that code that runs on 2.7 will run >on 2.8 unmodified. >>But a hypothetical 2.8 would also give people a way to move closer to >>py3k without giving up on using all their 2.x-only dependencies. > >How so? If they use anything that is new in 2.8, they *will* need to >drop support for anything before it, no??? >>I think it's much more likely that libraries like Twisted can support >>2.8 >>in the near future than 3.x. > >Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does >that help Twisted in moving to 3.2? I'm not reading this thread carefully enough to make any arguments on either side, but I can contribute a fact. Twisted very likely does not support 2.8 today. I base this on the fact that Twisted does not support 2.7 today, and I expect 2.8 will be more like 2.7 than it will be like 2.3 - 2.6 (which Twisted does support). When I say "support" here, I mean "all of the Twisted unit tests pass on it". Jean-Paul From tseaver at palladion.com Wed Jan 13 03:57:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jan 2010 21:57:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4D36AA.7020607@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karen Tracey wrote: > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> >> > Well yay. Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference. That's > down from 104 failures and 40 errors with 2.7 alpha 1. > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Just to add another "success" data point: Zope2's trunk, as well as the 2.12 release, passes all tests (2535 on the trunk) and brings up the appserver just fine under 2.7a2. There is an obnoxious deprecation warning out of the distutils: DeprecationWarning: 'compiler' specifies the compiler type in build_ext. If you want to get the compiler object itself, use 'compiler_obj' which is likely a simple one-line fix, if I only knew what the hell it was whining about. ;) The warning is extra obnoxious because it doesn't tell me what in *my* code triggers the warning (it (needs a 'stacklevel' argument). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktNNqQACgkQ+gerLs4ltQ7d5gCcCGKVjyOlxnrAln0UnRibS7kd lNIAoIs1RlSGMtJWaY11BqptfDmQvR87 =mIOO -----END PGP SIGNATURE----- From python at mrabarnett.plus.com Wed Jan 13 04:09:34 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 03:09:34 +0000 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D396E.4050500@mrabarnett.plus.com> MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > I've just noticed something odd about the re module: the sub() method doesn't take 'pos' or 'endpos' arguments. search() does; match() does; findall() does(); finditer() does; but sub() doesn't. Maybe there has never been a demand for it. (Nor split(), for that matter.) From brett at python.org Wed Jan 13 04:58:11 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 19:58:11 -0800 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On Tue, Jan 12, 2010 at 14:10, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > If it is a separate module under a different name it can do the proper thing. People will just need to be aware of the difference when they import the module. > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > That's totally up to you. There is practically no chance of it getting into the 2.x under the stdlib at this point since 2.7b1 is coming up and this module has not been out in the wild for a year (to my knowledge). If you want to support 2.x that's fine and I am sure users would appreciate it, but it isn't necessary to get into the Python 3 stdlib. > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > > If you want to minimize the differences then it should probably match. As I said, since it is a different name to import under it can deviate where reasonable, just make sure to clearly document the deviations. -Brett > Thanks > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jan 13 07:11:49 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 07:11:49 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D6425.3060307@v.loewis.de> > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) -1. AMD doesn't want us to use the term x86-64 anymore, but wants us to use AMD64 instead. I think we should comply - they invented the architecture, so they have the right to give it a name. Neither Microsoft nor Intel have such a right. Regards, Martin From sridharr at activestate.com Wed Jan 13 07:47:38 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 12 Jan 2010 22:47:38 -0800 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D6C8A.7070307@activestate.com> On 1/12/2010 2:46 PM, Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". FWIW, we simply use (64-bit, x64). Platform Download ============================================== Windows (x86) Windows Installer (MSI) Windows (64-bit, x64) Windows Installer (MSI) https://www.activestate.com/activepython/downloads/ -srid From solipsis at pitrou.net Wed Jan 13 08:08:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Jan 2010 07:08:55 +0000 (UTC) Subject: [Python-Dev] Fwd: Download Page - AMD64 References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) +1. I don't care about trademarks or official names, we should call it whatever is obvious for our users. As for Itanium, it is practically dead, I think there is little chance of people mixing it up with x86-64. cheers Antoine. From michael at voidspace.org.uk Wed Jan 13 10:32:00 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 09:32:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4D9310.6060907@voidspace.org.uk> On 13/01/2010 06:11, "Martin v. L?wis" wrote: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) >> >> instead of: >> >> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >> not include source) >> > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. > I think we should use whatever is most informative and least confusing to our users - we owe our allegiance to them and not to a processor vendor. Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Wed Jan 13 11:30:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:30:50 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF17A.1000503@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> Message-ID: <4B4DA0DA.7070007@gmail.com> Michael Foord wrote: > I agree with Martin that the *perception* is that to use Python 2.6 to > help you port to Python 3 you have to be willing to drop support for > earlier versions of Python. Note that increased 3.x compatibility in the most recent 2.x release will always help in two scenarios: 1. New projects that want to use 2.x only libraries but want to be ready for the Py3k transition in their own code (e.g. new 2.7 features like set literals, dict and set comprehensions and the view* dict methods can be translated far more cleanly by 2to3 than the closest comparable 2.6 code). 2. Similarly new projects that use a 3.x trunk can be more readily translated to a 2.7 version with 3to2, whereas some constructs may be difficult to recreate in earlier Python versions. I would expect this to be significantly less common then the first scenario though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:38:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:38:31 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4DA2A7.2080408@gmail.com> Antoine Pitrou wrote: > Christian Heimes cheimes.de> writes: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) > > +1. I don't care about trademarks or official names, we should call it whatever > is obvious for our users. Until this discussion, I didn't even know the architecture had any name other than x86-64. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:39:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:39:40 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4D36AA.7020607@palladion.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> Message-ID: <4B4DA2EC.3050908@gmail.com> Tres Seaver wrote: > There is an obnoxious deprecation warning out of the distutils: > > DeprecationWarning: 'compiler' specifies the compiler type in > build_ext. If you want to get the compiler object itself, use > 'compiler_obj' > > which is likely a simple one-line fix, if I only knew what the hell it > was whining about. ;) The warning is extra obnoxious because it doesn't > tell me what in *my* code triggers the warning (it (needs a 'stacklevel' > argument). Could you kick a tracker issues in Tarek's direction for that one? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Wed Jan 13 13:24:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 13 Jan 2010 12:24:09 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Brett Cannon python.org> writes: > If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. I'm sorry I won't be able to come to the language summit, but I would like if possible to expedite a pronouncement on PEP 391 (configuring logging using dictionaries). I believe I addressed all the comments made on the discussion threads mentioned in the PEP and so I'm not sure what more I need to do to get a pronouncement. I guess the stdlib slot gives an opportunity for people to air their views and so I'd be grateful if you added it to the agenda. Thanks & regards, Vinay Sajip From murman at gmail.com Wed Jan 13 14:35:51 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 07:35:51 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. I see and agree with the motivation behind your point, but it's just as reasonable to mimic the name Microsoft uses: the release is for Windows rather than the processor. On MSDN Microsoft uses parentheticals x86, ia64 and x64; this would suggest a name like Python 2.6.4 installer for Windows (x64). Unfortunately this usage doesn't seem to be reflected in consumer-oriented product pages, so I'm uncertain how clear it is for those downloading Python. -- Michael Urman From ncoghlan at gmail.com Wed Jan 13 15:04:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 00:04:28 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4DD2EC.3030709@gmail.com> Michael Urman wrote: > On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. > > I see and agree with the motivation behind your point, but it's just > as reasonable to mimic the name Microsoft uses: the release is for > Windows rather than the processor. On MSDN Microsoft uses > parentheticals x86, ia64 and x64; this would suggest a name like > Python 2.6.4 installer for Windows (x64). Unfortunately this usage > doesn't seem to be reflected in consumer-oriented product pages, so > I'm uncertain how clear it is for those downloading Python. As Windows doesn't run on non-x86 architectures, their naming is generally just Windows (32 bit) and Windows (64 bit). Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. In this case, making it clear that the 64-bit Windows binaries work for both Intel and AMD chips is more important than using only the official architecture name. Cheers, Nick. P.S. This discussion made me realise that out of habit I had dropped the 32-bit version of Python on my new 64-bit Windows box... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Wed Jan 13 17:49:55 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 13 Jan 2010 11:49:55 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA2EC.3050908@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> <4B4DA2EC.3050908@gmail.com> Message-ID: <4B4DF9B3.4030403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Tres Seaver wrote: >> There is an obnoxious deprecation warning out of the distutils: >> >> DeprecationWarning: 'compiler' specifies the compiler type in >> build_ext. If you want to get the compiler object itself, use >> 'compiler_obj' >> >> which is likely a simple one-line fix, if I only knew what the hell it >> was whining about. ;) The warning is extra obnoxious because it doesn't >> tell me what in *my* code triggers the warning (it (needs a 'stacklevel' >> argument). > > Could you kick a tracker issues in Tarek's direction for that one? Done: http://bugs.python.org/issue7694 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktN+bMACgkQ+gerLs4ltQ6s4gCdENhtVQWzoH449BCDz8SNx/4s DEoAn19LMcMs3b6ctdNF8K2hYd6d+BSZ =TWit -----END PGP SIGNATURE----- From rdmurray at bitdance.com Wed Jan 13 18:24:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Jan 2010 12:24:42 -0500 Subject: [Python-Dev] PYTHON3PATH Message-ID: <20100113172442.4A7771FA679@kimball.webabinitio.net> Please review issue 2375 [1], which is an enhancement request to add a PYTHON3PATH environment variable. Because we have elected to have both a python and a python3 command, I think this is an issue worth thinking about carefully to make sure we are serving the Python user community and easing the transition to python3. It could be that "use virtualenv" is the best answer, but I feel we should think about it carefully to make sure that is really true. [1] http://bugs.python.org/issue2375 -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From guido at python.org Wed Jan 13 18:31:39 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:31:39 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: Somehow the bug site doesn't load for me right now, but I'm -1 on this. There are maybe a dozen PYTHON... variables -- we really shouldn't try to add PYTHON3 variants for all of them. Specifically for PYTHONPATH, I feel that its use is always a short-time or localized hack, not something you set in your .profile nd forget about it (forgetting about it inparticular often leads to unnecessary debugging pain later). The better approaches are based on site-packages, wrapper scripts, virtualenv, etc. --Guido On Wed, Jan 13, 2010 at 9:24 AM, R. David Murray wrote: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. ?Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. ?It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 > > -- > R. David Murray ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dirkjan at ochtman.nl Wed Jan 13 18:38:26 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 13 Jan 2010 18:38:26 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 18:29, Benjamin Peterson wrote: > Please consider trying Python 2.7 with your code and reporting any bugs you may I ran the Mercurial test suite on current trunk, and it worked alright. There was a small issue with the fact that mimetypes got fooled by the lack of ImportError from _winreg (which I solved by blacklisting _winreg in demandimport), and one test fails likely because of a change in MIME handling. Will look into that. So the state of trunk looks rather solid from where I sit. Cheers, Dirkjan From ralf at brainbot.com Wed Jan 13 18:40:04 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:40:04 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> (R. David Murray's message of "Wed, 13 Jan 2010 12:24:42 -0500") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87r5ptdhu3.fsf@muni.brainbot.com> "R. David Murray" writes: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 The first thing I got while trying to run a python3 prompt few days ago, was an error. python3 tried to read my $PYTHONSTARTUP file, which used print statements. people will have to run both python 2 and python 3 code at the same time. Using different environment variables will make this easier. - Ralf From ralf at brainbot.com Wed Jan 13 18:55:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:55:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Guido van Rossum's message of "Wed, 13 Jan 2010 09:31:39 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87k4vldh4c.fsf@muni.brainbot.com> Guido van Rossum writes: > Somehow the bug site doesn't load for me right now, but I'm -1 on > this. There are maybe a dozen PYTHON... variables -- we really > shouldn't try to add PYTHON3 variants for all of them. > > Specifically for PYTHONPATH, I feel that its use is always a > short-time or localized hack, not something you set in your .profile > nd forget about it (forgetting about it inparticular often leads to > unnecessary debugging pain later). The better approaches are based on > site-packages, wrapper scripts, virtualenv, etc. then at least remove them completely from python3, so one can use them for his python 2 installation. Otherwise it will stump on some innocent python 3 program (and cause unnecessary debugging pain). From steven.bethard at gmail.com Wed Jan 13 18:57:29 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 13 Jan 2010 09:57:29 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: On Wed, Jan 13, 2010 at 9:40 AM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. ?Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. ?It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. How complicated is your PYTHONSTARTUP file? My suspicion is that you could easily write it to work for both Python 2.X and 3.X. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ssteinerx at gmail.com Wed Jan 13 18:57:42 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 13 Jan 2010 12:57:42 -0500 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> On Jan 13, 2010, at 12:40 PM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. Python 2 will continue to run as it has, and better solutions will emerge for Python 3 development. Beats the heck out of making a whole duplicate set for PYTHON3BLAHBLAH, yuck! S From guido at python.org Wed Jan 13 18:58:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:58:04 -0800 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: Memories of days past... Python had several regular expression implementations before, one of which was called "regex". But I would rather not have a new module -- I would much rather have a flag specifying the new (backwards incompatible) syntax/semantics. The flag would have a long name (e.g. re.NEW_SYNTAX), a short name (e.g. re.N) and an inline syntax, "(?n)...". --Guido On Tue, Jan 12, 2010 at 7:58 PM, Brett Cannon wrote: > > > On Tue, Jan 12, 2010 at 14:10, MRAB wrote: >> >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. >> > > If it is a separate module under a different name it can do the proper > thing. People will just need to be aware of the difference when they import > the module. > >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? >> > > That's totally up to you. There is practically no chance of it getting into > the 2.x under the stdlib at this point since 2.7b1 is coming up and this > module has not been out in the wild for a year (to my knowledge). ?If you > want to support 2.x that's fine and I am sure users would appreciate it, but > it isn't necessary to get into the Python 3 stdlib. > >> >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? >> > > If you want to minimize the differences then it should probably match. As I > said, since it is a different name to import under it can deviate where > reasonable, just make sure to clearly document the deviations. > -Brett > >> >> Thanks >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From ralf at brainbot.com Wed Jan 13 19:03:08 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 19:03:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Steven Bethard's message of "Wed, 13 Jan 2010 09:57:29 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <87fx69dgrn.fsf@muni.brainbot.com> Steven Bethard writes: > > How complicated is your PYTHONSTARTUP file? My suspicion is that you > could easily write it to work for both Python 2.X and 3.X. sure. that's exactly what I did. My point is that sharing those environment variables will cause pain for some people. From guido at python.org Wed Jan 13 19:07:58 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:07:58 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. -1. They have their use, but more in controlled situations. If you have "global" env vars that you only want to use with Python 2.x, write a wrapper for Python 3 that invokes it with -E. -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Wed Jan 13 19:14:21 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 13 Jan 2010 13:14:21 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E0D7D.3040806@scottdial.com> On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Wed Jan 13 19:22:33 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:22:33 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Wed, Jan 13, 2010 at 4:24 AM, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> If there is something missing from the stdlib discussion that you think should > be brought up at the summit let me know. And if there is something here you want > to discuss before the summit let me know and I can start a separate thread on it. > > I'm sorry I won't be able to come to the language summit, but I would like if > possible to expedite a pronouncement on PEP 391 (configuring logging using > dictionaries). I believe I addressed all the comments made on the discussion > threads mentioned in the PEP and so I'm not sure what more I need to do to get a > pronouncement. I guess the stdlib slot gives an opportunity for people to air > their views and so I'd be grateful if you added it to the agenda. Is the PEP in the stage of consensus yet? If so it may not even be necessary to discuss it at the summit (though I certainly won't stop people if they want to). -- --Guido van Rossum (python.org/~guido) From phd at phd.pp.ru Wed Jan 13 19:18:50 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 21:18:50 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <20100113181850.GA3837@phd.pp.ru> On Wed, Jan 13, 2010 at 12:57:42PM -0500, ssteinerX at gmail.com wrote: > Or, how about just removing the antiquated use of environment variables "antiquated"? You are kidding, aren't you?! Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Jan 13 19:51:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:51:14 -0800 Subject: [Python-Dev] PyCon Keynote Message-ID: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Jan 13 20:13:58 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:13:58 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D9310.6060907@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> Message-ID: <4B4E1B76.4010309@v.loewis.de> >>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>> X86-64 binary -- does not include source) >>> >>> instead of: >>> >>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>> not include source) >>> >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. >> > > I think we should use whatever is most informative and least confusing > to our users - we owe our allegiance to them and not to a processor vendor. And why do you think this is x86-64? Regards, Martin From martin at v.loewis.de Wed Jan 13 20:33:24 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:33:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA0DA.7070007@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> <4B4DA0DA.7070007@gmail.com> Message-ID: <4B4E2004.8050905@v.loewis.de> > Note that increased 3.x compatibility in the most recent 2.x release > will always help in two scenarios: > > 1. New projects that want to use 2.x only libraries but want to be ready > for the Py3k transition in their own code (e.g. new 2.7 features like > set literals, dict and set comprehensions and the view* dict methods can > be translated far more cleanly by 2to3 than the closest comparable 2.6 > code). Of these, I can only see this being the case of the view* methods. Why would set literals allow for code that more cleanly translates to 3.x? Suppose you write f = set((1,2,3)) in 2.6. That code will work *exactly* the same way in 3.1, no translation needed at all. Likewise for dict and set comprehensions: the code is as cleanly translated in the 2.7 form as it is in any prior form (ie. no modifications). As for view* methods: there is one important exception where the 2.6 equivalent is as cleanly translated as a view method: for key in D: action This is a more modern equivalent for iterating over D.keys(), so if your code still does the latter, just change it to the former (requires Python 2.2). I'd claim that this is the dominant case of traversing a dictionary (probably followed by iterating over .items()). So while it is true that only view* can be transformed correctly, I'd claim that this is an infrequent issue. > 2. Similarly new projects that use a 3.x trunk can be more readily > translated to a 2.7 version with 3to2, whereas some constructs may be > difficult to recreate in earlier Python versions. That may be true, but is besides the point here: the issue was whether a 2.8 release might help in migrating to 3.x. People who follow this approach already have 3.x code. Whether they would rather port to 2.8 only, and get that work with little effort, or whether they would port to 2.5 and later, and put in more work, I don't know - I guess we would have to ask these people. I would expect that they prefer if 2.x died rather sooner than later, because they can then stop supporting it. Regards, Martin From tjreedy at udel.edu Wed Jan 13 20:36:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Jan 2010 14:36:03 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: On 1/13/2010 2:13 PM, "Martin v. L?wis" wrote: >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. > > And why do you think this is x86-64? I do not believe I have personally seen, or at least noticed, that specific term before. Something like "Release for 64-bit Windows (Win NT, 2000, XP, Vista, 7 ) on AMD64-type processors from AMD and Intel" should be clear enough. From martin at v.loewis.de Wed Jan 13 20:40:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 20:40:30 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E21AE.40602@v.loewis.de> > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). Windows actually does - it runs on IA-64 (which is a non-x86 architecture). However, end users are unlikely to use such hardware, so distinguishing between 32-bit and 64-bit is typically good enough. > In this case, making it clear that the 64-bit Windows binaries work for > both Intel and AMD chips is more important than using only the official > architecture name. Well, we did have Itanium binaries at one point, and the "64-bit Windows binaries" you are referring to most definitely don't run on Itanium, even though that's an Intel chip. Regards, Martin From martin at v.loewis.de Wed Jan 13 20:45:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:45:40 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E0D7D.3040806@scottdial.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> Message-ID: <4B4E22E4.4040504@v.loewis.de> > So to echo what Michael said, the Microsoft nomenclature is "x64" > regardless of yours and Martin's objections to that name. Nobody who > uses Windows would be confused by "x64" since that is *the* Microsoft > naming scheme. That's actually not entirely true. There are several places in the APIs where Microsoft either allows or requires to call the architecture AMD64 (e.g. architecture indication in MSI files). Regards, Martin From regebro at gmail.com Wed Jan 13 20:50:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 20:50:59 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. What do you need to do in the PYTHONSTARTUP file? Ten years of Python programming, and I didn't even know it existed. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From phd at phd.pp.ru Wed Jan 13 21:08:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 23:08:40 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <20100113200840.GC14858@phd.pp.ru> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From murman at gmail.com Wed Jan 13 21:12:11 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 14:12:11 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E22E4.4040504@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> <4B4E22E4.4040504@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 13:45, "Martin v. L?wis" wrote: >> So to echo what Michael said, the Microsoft nomenclature is "x64" >> regardless of yours and Martin's objections to that name. Nobody who >> uses Windows would be confused by "x64" since that is *the* Microsoft >> naming scheme. > > That's actually not entirely true. There are several places in the > APIs where Microsoft either allows or requires to call the architecture > AMD64 (e.g. architecture indication in MSI files). I should have clarified I was talking about the names shown on MSDN subscriptions for downloading installation media of Windows 7 and Windows Vista. It's pretty clear in that context Microsoft uses x64 to describe this platform of Windows. But again, it's far from clear that this is a term they use for non-developers. -- Michael Urman From regebro at gmail.com Wed Jan 13 21:27:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 21:27:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113200840.GC14858@phd.pp.ru> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> Message-ID: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: > On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >> What do you need to do in the PYTHONSTARTUP file? >> Ten years of Python programming, and I didn't even know it existed. :-) > > ? See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Cheese and fries! :-) Anyway, I don't see how this could pose any problems, because any user advanced enough to do these things will be advanced enough to understand what the problem is and fix it so it works under Python 3 too. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ralf at brainbot.com Wed Jan 13 21:52:34 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 20:52:34 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> (Lennart Regebro's message of "Wed, 13 Jan 2010 20:50:59 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <874ompg225.fsf@brainbot.com> Lennart Regebro writes: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) hehe. tab completion: # -*- mode: python -*- # Last changed: 2009-12-23 22:25:15 by ralf import sys import os def initreadline(): try: import readline except ImportError: sys.stdout.write("Module readline not available.\n") return import rlcompleter readline.parse_and_bind("tab: complete") # Use tab for completions readline.parse_and_bind('tab: complete') # This forces readline to automatically print the above list when tab # completion is set to 'complete'. readline.parse_and_bind('set show-all-if-ambiguous on') # Bindings for incremental searches in the history. These searches # use the string typed so far on the command line and search # anything in the previous input history containing them. readline.parse_and_bind('"\C-r": reverse-search-history') readline.parse_and_bind('"\C-s": forward-search-history') history = os.path.expanduser("~/.pyhistory") if os.path.exists(history): readline.read_history_file(history) import atexit atexit.register(lambda: readline.write_history_file(history)) initreadline() del initreadline From martin at v.loewis.de Wed Jan 13 22:05:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 22:05:03 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <4B4E357F.4050107@v.loewis.de> Lennart Regebro wrote: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? I personally set up readline: set tab completion, and load the history of commands from the previous session. Of course, I don't know what Ralf uses it for. Regards, Martin From ncoghlan at gmail.com Wed Jan 13 22:43:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 07:43:41 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <4B4E3E8D.2010407@gmail.com> Guido van Rossum wrote: > On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about > just removing the antiquated use of environment variables altogether > from Python 3 and avoid the issue completely. > > -1. They have their use, but more in controlled situations. If you > have "global" env vars that you only want to use with Python 2.x, > write a wrapper for Python 3 that invokes it with -E. Perhaps a case can be made for Python 3 to assume -E by default, with a -e option to enable reading of the environment variables? That way naive users could run Python3 without tripping over existing Python2 environment variables, while other tools could readily set up a different environment before launching Python3. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Wed Jan 13 23:45:57 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 14:45:57 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 1:43 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users using environment variables? That's a recipe for disaster in any version! :-) -- --Guido van Rossum (python.org/~guido) From jared.grubb at gmail.com Wed Jan 13 23:56:37 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Wed, 13 Jan 2010 14:56:37 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <13F6C43A-D62A-4A9F-AF7A-9BDAC94F7D72@gmail.com> On 13 Jan 2010, at 13:43, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. I raised a question about these PYTHON3* variables once before in a discussion about shebang lines: http://www.mailinglistarchive.com/python-dev at python.org/msg29500.html I'm not advocating them, but just wanted to make sure to bring up the shebang use case. Jared From skip at pobox.com Wed Jan 13 23:24:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 16:24:13 -0600 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <19278.18445.428716.321365@montanaro.dyndns.org> Lennart> What do you need to do in the PYTHONSTARTUP file? Just reading off stuff from my own personal startup file... I use it for stuff I want available during interactive sessions: 1. Enable true division. 2. Conditionally define "help" from back in the days when there was no help builtin function. 3. Auto-save session (readline) history so I can easily recall commands across sessions. 4. Add other fake builtins ("like") or override behavior of some (like "dir") making them handier for interactive use. 5. autoload modules/symbols (pokes around in common modules from sys.excepthook function). Oh, and I've had no particular trouble keeping it working in Python 1, 2 or 3. Skip From fuzzyman at voidspace.org.uk Thu Jan 14 01:20:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 00:20:21 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: <4B4E6345.5070202@voidspace.org.uk> On 13/01/2010 19:13, "Martin v. L?wis" wrote: >>>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>>> X86-64 binary -- does not include source) >>>> >>>> instead of: >>>> >>>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>>> not include source) >>>> >>>> >>> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >>> to use AMD64 instead. I think we should comply - they invented the >>> architecture, so they have the right to give it a name. Neither >>> Microsoft nor Intel have such a right. >>> >>> >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. >> > And why do you think this is x86-64? > Well anecdotal everyone I have *every* talked to about 64bit processors has referred to having a 64bit processor (x86 is a given) and not an AMD64 architecture processor. Linus Torvalds addressed this specific issue for Linux and came down on the side of "x86-64": http://kerneltrap.org/node/2466 Look up AMD64 on Wikipedia and it redirects you to the X86-64 page. Information website setup by AMD and partners about the AMD64 architecture: http://www.x86-64.org/about.html In the AMD website they refer to "x86-64 Assembly": http://www.x86-64.org/documentation/assembly.html Microsoft seem to draw a distinction between x64 (which would also be acceptable) and Itanium based systems. Very rarely do they refer to AMD64: * http://www.microsoft.com/servers/64bit/compare.mspx * http://www.microsoft.com/servers/64bit/x64/overview.mspx * http://www.microsoft.com/servers/64bit/overview.mspx Using a vendor specific name automatically begs the question as to whether the installer works on processors from other vendors, as we saw in the specific enquiry from the user that triggered this debate. Referring to the AMD 64 build as x86-64, with a footnote explaining which architectures this specifically means is unlikely to confuse people. It is *definitely* better than just saying AMD64. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Thu Jan 14 02:50:55 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 19:50:55 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? Message-ID: <19278.30847.649228.115514@montanaro.dyndns.org> Just out of curiosity, is the static build stuff (use the old Modules/Setup file to build modules) exercised at all any more? Skip From benjamin at python.org Thu Jan 14 03:22:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 13 Jan 2010 20:22:03 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? In-Reply-To: <19278.30847.649228.115514@montanaro.dyndns.org> References: <19278.30847.649228.115514@montanaro.dyndns.org> Message-ID: <1afaf6161001131822r151bd202x35653ba44ee0235d@mail.gmail.com> 2010/1/13 : > > Just out of curiosity, is the static build stuff (use the old Modules/Setup > file to build modules) exercised at all any more? Exercised as in used in the build process? Yes. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Jan 14 10:23:47 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 09:23:47 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! Message-ID: <954450.26617.qm@web25804.mail.ukl.yahoo.com> In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: http://www.python.org/dev/peps/pep-0391/ In November 2009 I posted to this list that the PEP was ready for review. I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. Thanks & regards, Vinay Sajip From ziade.tarek at gmail.com Thu Jan 14 10:30:15 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Jan 2010 10:30:15 +0100 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <94bdd2611001140130u6154e893jfd37db32a25be66a@mail.gmail.com> On Thu, Jan 14, 2010 at 10:23 AM, Vinay Sajip wrote: [..] > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would > be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check > in the changes unless there are objections to doing so. All those who think that logging is currently > hard to configure will benefit from these changes, so here's the opportunity to pipe up and > improve things. FWIW, I am +1. Thanks for your work Tarek From hodgestar+pythondev at gmail.com Thu Jan 14 10:39:41 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 14 Jan 2010 11:39:41 +0200 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: On Thu, Jan 14, 2010 at 11:23 AM, Vinay Sajip wrote: > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I think the dict configuration scheme will be a great addition to the logging module. :) Schiavo Simon From p.f.moore at gmail.com Thu Jan 14 12:22:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jan 2010 11:22:09 +0000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> 2010/1/14 Vinay Sajip : > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I've no immediate need for the feature, but it would be good to have something like this, so I'm +1. Paul. From ncoghlan at gmail.com Thu Jan 14 12:46:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:46:19 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> Message-ID: <4B4F040B.8010607@gmail.com> Paul Moore wrote: > 2010/1/14 Vinay Sajip : >> So, can you please indicate your vote for or against incorporating PEP 391 into Python? > > I've no immediate need for the feature, but it would be good to have > something like this, so I'm +1. I'm in the same boat as Paul, but PEP 291 provides a nice forward compatible configuration scheme that will work with any application configuration approach that can produce an appropriate Python dictionary. So +1 from me too - I think the PEP has now taken this as far as theorising can go, and the only way to learn anything further is to put it into practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jan 14 12:57:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:57:55 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4B4F06C3.7030806@gmail.com> Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of > configuring logging using dictionaries: > > http://www.python.org/dev/peps/pep-0391/ Although one minor comment: you can probably remove the note about the "ext://" and "cfg://" prefixes being provisional at this stage. Those names look fine to me, so I don't see any point inviting a late-breaking bikeshed discussion about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu Jan 14 14:53:29 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 08:53:29 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> On Thu, Jan 14, 2010 at 4:23 AM, Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: > > ?http://www.python.org/dev/peps/pep-0391/ > > In November 2009 I posted to this list that the PEP was ready for review. > > I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. > > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. > > Thanks & regards, > > Vinay Sajip I'm generally +1 - but given I know that Django 1.2 is slated to implement something somewhat similar, I'm interested to hear how this proposal meshes with their plan(s). jesse From vinay_sajip at yahoo.co.uk Thu Jan 14 15:08:24 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 14:08:24 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> Message-ID: <92210.91656.qm@web25806.mail.ukl.yahoo.com> > From: Jesse Noller > I'm generally +1 - but given I know that Django 1.2 is slated to > implement something somewhat similar, I'm interested to hear how this > proposal meshes with their plan(s).. Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: http://groups.google.com/group/django-developers/msg/4ef81a2160257221 They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. Regards, Vinay Sajip From ssteinerx at gmail.com Thu Jan 14 15:54:27 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 14 Jan 2010 09:54:27 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <62E362ED-5DD7-4D42-B5E0-B263A137FD5C@gmail.com> On Jan 14, 2010, at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. That puts a huge +1 on there for me. If we can get this approved and have Django's logging improved *and* avoid a whole pile of duplicate effort in Django that's a huge win. S From jnoller at gmail.com Thu Jan 14 15:56:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 09:56:18 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <4222a8491001140656w68c7290agccfe7282cf96f2fb@mail.gmail.com> On Thu, Jan 14, 2010 at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. > > Regards, > > Vinay Sajip > Cool, +1 then :) From mal at egenix.com Thu Jan 14 20:19:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 20:19:12 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <4B4F6E30.50400@egenix.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users won't have PYTHONPATH or any other Python environment variables setup. Really, if you know that you are going to run Python 3 instead of Python 2 or vice-versa it's easy enough to run . py3env.sh or . py2env.sh in order to setup your shell environment, much like you typically do when using virtual Python installations or work on different projects that require different setups. If you just want to separate Python 2 and 3 files, use the user site-packages dir which includes the Python version. More experienced users could also write their own environment switching sitecustomize.py or usercustomize.py. And then there's the hackery option for experts that love dirty tricks: add a .pth file which includes an "import mysetup" line to fix-up you path and other settings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jan 14 22:02:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Jan 2010 07:02:28 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <4B4F8664.4080107@gmail.com> M.-A. Lemburg wrote: > Nick Coghlan wrote: >> Guido van Rossum wrote: >>> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >>> just removing the antiquated use of environment variables altogether >>> from Python 3 and avoid the issue completely. >>> >>> -1. They have their use, but more in controlled situations. If you >>> have "global" env vars that you only want to use with Python 2.x, >>> write a wrapper for Python 3 that invokes it with -E. >> Perhaps a case can be made for Python 3 to assume -E by default, with a >> -e option to enable reading of the environment variables? >> >> That way naive users could run Python3 without tripping over existing >> Python2 environment variables, while other tools could readily set up a >> different environment before launching Python3. > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. I was actually thinking of the situation where the OS had a preinstalled Python 2 with a default PYTHONPATH setting and the user was playing with Python 3. However, I agree that that is a fairly unlikely scenario (since preinstalled Pythons tend not to rely on the environment variables and a user sophisticated enough to be playing with a new Python interpreter should be able to cope with a few environment variable conflicts anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Thu Jan 14 22:09:30 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 21:09:30 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F8664.4080107@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <4B4F8664.4080107@gmail.com> Message-ID: <4B4F880A.5010809@voidspace.org.uk> On 14/01/2010 21:02, Nick Coghlan wrote: > However, I agree that that is a fairly unlikely scenario (since > preinstalled Pythons tend not to rely on the e Well, on the other hand I think that during the next few years it will be increasingly common for developers (and possibly users) to have Python 2 and Python 3 installed side-by-side. Many libraries and applications may never make the jump to Python 3 and Python users may be using 'legacy' Python 2 code for many years to come. It will also become increasingly common for developers to be using Python 3 *primarily* and for Python 3 only libraries and applications to emerge. Whilst there are workarounds we *are* in a situation that Python 2 and Python 3 share environment variables for the location of libraries and executing code on startup, whilst at the same time they are largely incompatible and need separate library paths and startup code. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ralf at brainbot.com Thu Jan 14 22:25:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Thu, 14 Jan 2010 22:25:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> (M.'s message of "Thu, 14 Jan 2010 20:19:12 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <871vhswf90.fsf@brainbot.com> "M.-A. Lemburg" writes: > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. > > Really, if you know that you are going to run Python 3 instead of > Python 2 or vice-versa it's easy enough to run You don't even know that you're going to run python. I have 40 python scripts in my /usr/bin directory. > > . py3env.sh > or > . py2env.sh > > in order to setup your shell environment, much like you typically > do when using virtual Python installations or work on different > projects that require different setups. No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > > If you just want to separate Python 2 and 3 files, use the > user site-packages dir which includes the Python version. Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to install those programs in a user's home directory. There are still people running python <2.6. From mal at egenix.com Thu Jan 14 22:51:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 22:51:07 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <871vhswf90.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <871vhswf90.fsf@brainbot.com> Message-ID: <4B4F91CB.2060106@egenix.com> Ralf Schmitt wrote: > "M.-A. Lemburg" writes: > >> >> Naive users won't have PYTHONPATH or any other Python environment >> variables setup. >> >> Really, if you know that you are going to run Python 3 instead of >> Python 2 or vice-versa it's easy enough to run > > You don't even know that you're going to run python. I have 40 python > scripts in my /usr/bin directory. > >> >> . py3env.sh >> or >> . py2env.sh >> >> in order to setup your shell environment, much like you typically >> do when using virtual Python installations or work on different >> projects that require different setups. > > No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > >> >> If you just want to separate Python 2 and 3 files, use the >> user site-packages dir which includes the Python version. > > Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to > install those programs in a user's home directory. There are still > people running python <2.6. Note that you only need to have a different PYTHONPATH setups for Python 2.x/3.x if you plan to run code that: * is not installed in site-packages (most OS shipped code will be found in the resp. system site-packages/ dir) * is not installed in a user site-packages directory (user installed code will typically go there (*)) * uses modules/packages that come in two different versions (one for Python 2.x and one for 3.x) and use the same name for both versions * needs to work in both Python 2.x and 3.x (*) The method for installing code in user site-packages dir is running: python setup.py install --user instead of the standard python setup.py install which install to the system-wide site-packages. BTW: I guess the bzr and mercurial wikis will need to be updated accordingly - at least for users of Python >=2.6. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Jan 14 22:55:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jan 2010 22:55:04 +0100 Subject: [Python-Dev] [ANN] Python 2.5.5 Release Candidate 1. Message-ID: <4B4F92B8.30806@v.loewis.de> On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.5. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.6 (which is 2.6.4 at this point). This releases fixes issues with the logging and tarfile modules, and with thread-local variables. See the detailed release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed. For more information on Python 2.5.5, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.5.5 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin at v.loewis.de Python Release Manager (on behalf of the entire python-dev team) From status at bugs.python.org Fri Jan 15 18:07:26 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Jan 2010 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100115170726.9963A7865F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/08/10 - 01/15/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2536 open (+32) / 16993 closed (+16) / 19529 total (+48) Open issues with patches: 1024 Average duration of open issues: 710 days. Median duration of open issues: 469 days. Open Issues Breakdown open 2501 (+32) pending 34 ( +0) Issues Created Or Reopened (53) _______________________________ PYTHON3PATH environment variable to supersede PYTHONPATH for mul 01/13/10 CLOSED http://bugs.python.org/issue2375 reopened r.david.murray Mark the compiler package as deprecated 01/13/10 http://bugs.python.org/issue6837 reopened ezio.melotti test_distutils failure 01/15/10 CLOSED http://bugs.python.org/issue6961 reopened flox buildbot test_urllib: unsetting missing 'env' variable 01/08/10 CLOSED http://bugs.python.org/issue7026 reopened flox Two float('nan') are not equal 01/08/10 CLOSED http://bugs.python.org/issue7660 created jmfauth compiling ctypes fails with non-ascii path 01/08/10 http://bugs.python.org/issue7661 created pitrou patch time.utcoffset() 01/09/10 http://bugs.python.org/issue7662 created techtonik UCS4 build incorrectly translates cases for non-BMP code points 01/10/10 CLOSED http://bugs.python.org/issue7663 created exarkun python logger does not handle IOError Exception 01/10/10 CLOSED http://bugs.python.org/issue7664 created yateenjoshi test_urllib2 and test_ntpath fail if path contains "\" 01/10/10 http://bugs.python.org/issue7665 created flox test_lib2to3 fails if path contains space 01/10/10 CLOSED http://bugs.python.org/issue7666 created flox test_doctest fails with non-ascii path 01/10/10 http://bugs.python.org/issue7667 created flox buildbot test_httpservers fails with non-ascii path 01/10/10 http://bugs.python.org/issue7668 created flox buildbot test_unicode_file fails with non-ascii path 01/10/10 CLOSED http://bugs.python.org/issue7669 created flox _sqlite3: Block *all* operations on a closed Connection object 01/10/10 http://bugs.python.org/issue7670 created haypo patch test_popen fails if path contains special char like ";" 01/11/10 http://bugs.python.org/issue7671 reopened flox patch _ssl module causes segfault 01/10/10 http://bugs.python.org/issue7672 created ssoria patch audioop: check that length is a multiple of the size 01/11/10 http://bugs.python.org/issue7673 created haypo patch select.select() corner cases: duplicate fds, out-of-range fds 01/11/10 http://bugs.python.org/issue7674 created cdleary help() doesn't accept unicode literals in built in docstrings 01/11/10 http://bugs.python.org/issue7675 created psd patch IDLE shell shouldn't use TABs 01/11/10 http://bugs.python.org/issue7676 created cben distutils, better error message for setup.py upload -sign withou 01/11/10 http://bugs.python.org/issue7677 created illume subprocess.Popen pipeline example code in the documentation is l 01/11/10 http://bugs.python.org/issue7678 created steven.k.wong Warning building 2.7 on OS X 10.6 libintl.h "Present But Cannot 01/12/10 http://bugs.python.org/issue7679 created ssteinerX pythonw crash while attempting to start() a thread object 01/12/10 http://bugs.python.org/issue7680 created dontbugme Incorrect division in [wave.py] 01/12/10 CLOSED http://bugs.python.org/issue7681 created alfps patch, needs review Optimisation of if with constant expression 01/12/10 http://bugs.python.org/issue7682 created sdefresne patch Wrong link in HTML documentation 01/12/10 CLOSED http://bugs.python.org/issue7683 created francescor decimal.py: infinity coefficients in tuples 01/12/10 http://bugs.python.org/issue7684 created skrah minor typo in re docs 01/12/10 CLOSED http://bugs.python.org/issue7685 created mikejs patch redundant open modes 'rbb', 'wbb', 'abb' no longer work on Windo 01/13/10 http://bugs.python.org/issue7686 created ivank Bluetooth support untested 01/13/10 http://bugs.python.org/issue7687 created pitrou TypeError: __name__ must be set to a string object 01/13/10 http://bugs.python.org/issue7688 created frankmillman Pickling of classes with a metaclass and copy_reg 01/13/10 http://bugs.python.org/issue7689 created craigcitro patch Wrong PEP number in importlib module manual page 01/13/10 CLOSED http://bugs.python.org/issue7690 created francescor test_bsddb3 files are not always removed when test fails 01/13/10 CLOSED http://bugs.python.org/issue7691 created flox buildbot Pointless comparision of unsigned integer with zero 01/13/10 CLOSED http://bugs.python.org/issue7692 created Bugger tarfile.extractall can't have unicode extraction path 01/13/10 http://bugs.python.org/issue7693 created pbienst DeprecationWarning from distuils.commands.build_ext needs stackl 01/13/10 http://bugs.python.org/issue7694 created tseaver patch missing termios constants 01/13/10 http://bugs.python.org/issue7695 created conorh Improve Memoryview/Buffer documentation 01/13/10 http://bugs.python.org/issue7696 created flox os.getcwd() should raise UnicodeDecodeError for arbitrary bytes 01/13/10 CLOSED http://bugs.python.org/issue7697 created flox pystack macro in Misc/gdbinit incorrectly uses PyEval_EvalFrame 01/14/10 CLOSED http://bugs.python.org/issue7698 created exarkun patch strptime, strftime documentation 01/14/10 http://bugs.python.org/issue7699 created mikejs patch "-3" flag does not work anymore 01/14/10 CLOSED http://bugs.python.org/issue7700 created flox patch fix output string length for binascii.b2a_uu() 01/14/10 CLOSED http://bugs.python.org/issue7701 created haypo patch Wrong order of parameters of _get_socket in SMTP class in smtpli 01/14/10 http://bugs.python.org/issue7702 created alito Replace buffer()-->memoryview() in Lib/ctypes/test/ 01/14/10 http://bugs.python.org/issue7703 created flox patch Math calculation problem (1.6-1.0)>0.6, python said TRUE 01/14/10 CLOSED http://bugs.python.org/issue7704 created DhaReaL libpython2.6.so is not linked correctly on FreeBSD when threads 01/15/10 http://bugs.python.org/issue7705 created aballier patch, needs review Missing #include guards 01/15/10 http://bugs.python.org/issue7706 created akrpic77 patch multiprocess.Queue operations during import can lead to deadlock 01/15/10 http://bugs.python.org/issue7707 created kripken Conversion of longs to bytes and vice-versa. 01/11/10 http://bugs.python.org/issue1023290 reopened mark.dickinson patch Issues Now Closed (67) ______________________ segfault in curses when calling redrawwin() before refresh() 825 days http://bugs.python.org/issue1266 mark.dickinson "RuntimeError: dictionary changed size during iteration" in Tkin 751 days http://bugs.python.org/issue1658 flox patch Exception exceptions.AttributeError '_shutdown' in 0.6, python said TRUE 0 days http://bugs.python.org/issue7704 tim_one Support for sending multipart form data 2452 days http://bugs.python.org/issue727898 r.david.murray email.MIME*.as_string removes duplicate spaces 1440 days http://bugs.python.org/issue1422094 r.david.murray easy test_parsedate_acceptable_to_time_functions+DST == :-( 1394 days http://bugs.python.org/issue1454285 r.david.murray email module does not complay with RFC 2046: CRLF issue 1196 days http://bugs.python.org/issue1571841 r.david.murray email.FeedParser.BufferedSubFile improperly handles "\r\n" 969 days http://bugs.python.org/issue1721862 r.david.murray patch, easy Top Issues Most Discussed (10) ______________________________ 20 dtoa.c: oversize b in quorem 11 days open http://bugs.python.org/issue7632 18 _ssl module causes segfault 5 days open http://bugs.python.org/issue7672 12 Speed up cPickle's pickling generally 287 days open http://bugs.python.org/issue5683 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 7 days open http://bugs.python.org/issue7658 8 Fatal error on thread creation in low memory condition 27 days open http://bugs.python.org/issue7544 8 Direct calls to PyObject_Del/PyObject_DEL are broken for --with 558 days open http://bugs.python.org/issue3299 7 "-3" flag does not work anymore 1 days closed http://bugs.python.org/issue7700 7 tarfile.extractall can't have unicode extraction path 2 days open http://bugs.python.org/issue7693 7 test_urllib: unsetting missing 'env' variable 0 days closed http://bugs.python.org/issue7026 7 os.path.abspath with unicode argument should call os.getcwdu 542 days open http://bugs.python.org/issue3426 From g.brandl at gmx.net Sat Jan 16 21:05:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 16 Jan 2010 21:05:46 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> Message-ID: Am 13.01.2010 21:27, schrieb Lennart Regebro: > On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: >> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >>> What do you need to do in the PYTHONSTARTUP file? >>> Ten years of Python programming, and I didn't even know it existed. :-) >> >> See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. > > Cheese and fries! :-) > > Anyway, I don't see how this could pose any problems, because any user > advanced enough to do these things will be advanced enough to > understand what the problem is and fix it so it works under Python 3 > too. I'd propose adding a bit of text to the environment variables documentation, and especially about the needs of PYTHONSTARTUP files. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From asmodai at in-nomine.org Sat Jan 16 21:50:56 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 16 Jan 2010 21:50:56 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <874ompg225.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> Message-ID: <20100116205056.GL99670@nexus.in-nomine.org> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >hehe. tab completion: With bpython and ipython available, why would you even want to stick to the 'plain old' interactive interpreter? (Sorry to derail the discussion, but maybe there's more people that have not heard of either or both.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B For ever, brother, hail and farewell... From solipsis at pitrou.net Sat Jan 16 21:57:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Jan 2010 20:57:13 +0000 (UTC) Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: Jeroen Ruigrok van der Werven in-nomine.org> writes: > > -On [20100113 22:13], Ralf Schmitt (ralf brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick to the > 'plain old' interactive interpreter? Why wouldn't we? There are probably many more people using the standard Python prompt than ipython. From ben+python at benfinney.id.au Sat Jan 16 23:41:03 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Jan 2010 09:41:03 +1100 Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: <87y6jxk70g.fsf@benfinney.id.au> Jeroen Ruigrok van der Werven writes: > -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick > to the 'plain old' interactive interpreter? Because those optional extras are not always available, whereas the standard interactive interpreter is always available with CPython distributions. -- \ ?I fly Air Bizarre. You buy a combination one-way round-trip | `\ ticket. Leave any Monday, and they bring you back the previous | _o__) Friday. That way you still have the weekend.? ?Steven Wright | Ben Finney From ncoghlan at gmail.com Sun Jan 17 01:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:22:10 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87y6jxk70g.fsf@benfinney.id.au> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> <87y6jxk70g.fsf@benfinney.id.au> Message-ID: <4B525832.8090904@gmail.com> Ben Finney wrote: > Jeroen Ruigrok van der Werven writes: > >> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >>> hehe. tab completion: >> With bpython and ipython available, why would you even want to stick >> to the 'plain old' interactive interpreter? > > Because those optional extras are not always available, whereas the > standard interactive interpreter is always available with CPython > distributions. I've never even contemplated the idea of installing 3rd party apps for the 4 current development builds (2.x trunk, 2.x maintenance, 3.x trunk, 3.x maintenance) on my home development machine. And of course work suffers from the problem of not being allowed to install arbitrary apps I downloaded from the internet without getting the licensing vetted for commercial acceptability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 01:34:08 2010 From: nad at acm.org (Ned Deily) Date: Sat, 16 Jan 2010 16:34:08 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? Message-ID: I've recently seen a couple of references to 3.1.2 go by in checkins which made me wonder whether dates have been proposed yet for updates to either 3.1 or 2.6. I don't recall seeing any and I didn't see any references in the PEPs. Some advance warning would be nice. I assume that some critical problem could trigger planning for an update on short notice; is there a time-limit trigger as well? -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Jan 17 01:55:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:55:38 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52600A.7060201@gmail.com> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Usually there is one more regular maintenance release of the existing maintenance branches within a few months of the creation of the next version (releases before then are at the discretion of the release manager, and security releases continue for much longer). So take the planned 2.7 and 3.2 release dates and add a couple of months to each one to get the likely release dates for the 2.6.x and 3.1.x maintenance releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sun Jan 17 10:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 17 Jan 2010 10:21:49 +0100 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52D6AD.6090302@v.loewis.de> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Barry was once proposing time-based releases; this idea didn't really catch on. It's the release manager who decides when the next bug fix release will be made, often in response to developers asking for one. Regards, Martin From solipsis at pitrou.net Sun Jan 17 14:00:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 13:00:04 +0000 (UTC) Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: Message-ID: Ned Deily acm.org> writes: > > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. There are a couple of release blockers right now. Once they are fixed or deferred, I think it would be nice to have a 3.1.2. Why do you need "some advance warning" though? From ncoghlan at gmail.com Sun Jan 17 14:40:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 23:40:42 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B53135A.7060104@gmail.com> Antoine Pitrou wrote: > Ned Deily acm.org> writes: >> I've recently seen a couple of references to 3.1.2 go by in >> checkins which made me wonder whether dates have been proposed yet >> for updates to either 3.1 or 2.6. I don't recall seeing any and I >> didn't see any references in the PEPs. Some advance warning would >> be nice. > > There are a couple of release blockers right now. Once they are fixed > or deferred, I think it would be nice to have a 3.1.2. Why do you > need "some advance warning" though? Advance warning does allow interested users that would consider upgrading to schedule time for testing before the maintenance release comes out. This is particularly useful in helping to make a 1-week RC period effective in picking up issues that might otherwise lead to a brown paper bag release to fix major issues that slipped through our own automated test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 19:01:37 2010 From: nad at acm.org (Ned Deily) Date: Sun, 17 Jan 2010 10:01:37 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: <4B53135A.7060104@gmail.com> Message-ID: In article <4B53135A.7060104 at gmail.com>, Nick Coghlan wrote: > Antoine Pitrou wrote: > > Ned Deily acm.org> writes: > >> I've recently seen a couple of references to 3.1.2 go by in > >> checkins which made me wonder whether dates have been proposed yet > >> for updates to either 3.1 or 2.6. I don't recall seeing any and I > >> didn't see any references in the PEPs. Some advance warning would > >> be nice. > > There are a couple of release blockers right now. Once they are fixed > > or deferred, I think it would be nice to have a 3.1.2. Why do you > > need "some advance warning" though? > > Advance warning does allow interested users that would consider > upgrading to schedule time for testing before the maintenance release > comes out. This is particularly useful in helping to make a 1-week RC > period effective in picking up issues that might otherwise lead to a > brown paper bag release to fix major issues that slipped through our own > automated test coverage. That. and resource contention: there are always potential fixes in the pipeline that could or should be bumped in priority if one knows there is a code cutoff approaching. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Sun Jan 17 20:51:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 20:51:58 +0100 Subject: [Python-Dev] Enhancing the shutil module Message-ID: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Hello, For 2.7/3.2, I am in the process of removing modules in Distutils that can be replaced by calls to existing functions in stdlib. For instance, "dir_util" and "file_util" (old modules from the Python 1.x era) are going away in favor of calls to shutil (and os), so the Distutils package gets lighter. Another module I would like to move away from Distutils is "archive_util". It contains helpers to build archives, whether they are zip or tar files. I propose to move those useful functions into shutil, as this seems the most logical place. I also propose to maintain this "shutil" module for now on (no one is declared as a maintainer in maintainers.rst) since Distutils will become a heavy user of its functions. Any objections/opinions ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 20:53:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 19:53:03 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536A9F.5050203@voidspace.org.uk> On 17/01/2010 19:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > > Regards, > Tarek > > I think it's a great idea. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Sun Jan 17 20:55:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2010 11:55:47 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > If it's archive-agnostic then shutil is probably the best place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Great! > Any objections/opinions ? > No objections from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jan 17 21:04:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 20:04:41 +0000 (UTC) Subject: [Python-Dev] Enhancing the shutil module References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: Tarek Ziad? gmail.com> writes: > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. Are these functions portable? Do they rely on external programs? > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. It's ok for me. Regards Antoine. From ziade.tarek at gmail.com Sun Jan 17 21:09:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:09:18 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: > > > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >> >> Hello, >> >> For 2.7/3.2, I am in the process of removing modules in Distutils that >> can be replaced by calls to existing functions in stdlib. For >> instance, "dir_util" and "file_util" (old modules from the Python 1.x >> era) are going away in favor of calls to shutil (and os), so the >> Distutils package gets lighter. >> >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. > > If it's archive-agnostic then shutil is probably the best place. In more details: It allows the creation of gzip, bzip2, tar and zip files through a single API. There's a registry of supported formats and the API is driven by a format identifier. To do the work it uses stdlib's compression modules. Although it tries the "zip" system command as a fallback if the "zipfile" module is not present. (notice that I've removed the support of "compress" (.Z) some time ago) Regards Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 21:15:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 20:15:00 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536FC4.9090304@voidspace.org.uk> On 17/01/2010 20:04, Antoine Pitrou wrote: > Tarek Ziad? gmail.com> writes: > >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. >> > Are these functions portable? Do they rely on external programs? > > I believe that part of the work that Tarek has been doing has been to make these distutils commands use the Python standard library and not depend on external programs. In which case they seem like *excellent* additions to the shutil module. Of course Tarek can speak for himself... Michael >> I also propose to maintain this "shutil" module for now on (no one is >> declared as a maintainer in maintainers.rst) since Distutils will >> become a heavy user of its functions. >> > It's ok for me. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Sun Jan 17 21:43:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:43:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B536FC4.9090304@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> Message-ID: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> On Sun, Jan 17, 2010 at 9:15 PM, Michael Foord wrote: [..] >> Are these functions portable? Do they rely on external programs? >> >> > > I believe that part of the work that Tarek has been doing has been to make > these distutils commands use the Python standard library and not depend on > external programs. In which case they seem like *excellent* additions to the > shutil module. yes, in the past the "tar" files where created using the "tar" command but this has been changed. For a while now, they are portable and they use stdlib code only. A recent addition is to be able to define user/group permissions in the tar files, thanks to Lars' work in the tarfile module. There's one remaining external call for "zip" done if the zip module is not found, but I am happy to remove it and throw an exception if it's not found, and keep the external "zip" call on Distutils side, so shutil stays 100% stdlib-powered. > Of course Tarek can speak for himself... Thanks for explaining it ! :) Regards Tarek -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Sun Jan 17 22:50:52 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sun, 17 Jan 2010 13:50:52 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> Message-ID: <4B53863C.5060304@activestate.com> On 1/17/2010 12:09 PM, Tarek Ziad? wrote: > On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: >> > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >>> >> Another module I would like to move away from Distutils is >>> >> "archive_util". It contains helpers to build archives, whether they >>> >> are zip or tar files. I propose to move those useful functions into >>> >> shutil, as this seems the most logical place. >> > If it's archive-agnostic then shutil is probably the best place. > In more details: > It allows the creation of gzip, bzip2, tar and zip files through a single API. > There's a registry of supported formats and the API is driven by a > format identifier. Will it also allow decompression of the said archive types? Distribute has some utility code to handle zip/tar archives. So does PyPM. This is because the `tarfile` and `zipfile` modules do not "just work" due to several issues. See http://gist.github.com/279606 Take note of the following in the above code: 1) _ensure_read_write_access 2) *File.is_valid 3) ZippedFile.extract ... issue 6510 4) ZippedFile.extract ... issue 6609 5) TarredFile.extract ... issue 6584 6) The way unpack() detects the unpacked directory. -srid From ziade.tarek at gmail.com Sun Jan 17 23:09:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 23:09:29 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B53863C.5060304@activestate.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> <4B53863C.5060304@activestate.com> Message-ID: <94bdd2611001171409j4ec1e0bfj47e59894fdec0d8a@mail.gmail.com> On Sun, Jan 17, 2010 at 10:50 PM, Sridhar Ratnakumar wrote: [..] > Will it also allow decompression of the said archive types? No but it would make sense having this one as well. Distribute/Setuptools' "unpack_archive" (used by easy_install) was implemented using the same principle as Distutils' "make_archive". I can add it as well while I am at it : it has been successfully used for years by easy_install. > Distribute has > some utility code to handle zip/tar archives. So does PyPM. This is because > the `tarfile` and `zipfile` modules do not "just work" due to several > issues. > > See http://gist.github.com/279606 > > Take note of the following in the above code: > > ?1) _ensure_read_write_access > ?2) *File.is_valid > ?3) ZippedFile.extract ... issue 6510 > ?4) ZippedFile.extract ... issue 6609 > ?5) TarredFile.extract ... issue 6584 Looks like some of these are already fixed (I just looked quickly in the bug tracker though) If its not already done and pending, it would be great if you could refactor your fixes into patches for the remaining issues for each one of those modules Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Sun Jan 17 23:56:37 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 17 Jan 2010 17:56:37 -0500 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: <4B52D6AD.6090302@v.loewis.de> References: <4B52D6AD.6090302@v.loewis.de> Message-ID: On Jan 17, 2010, at 4:21 AM, Martin v. L?wis wrote: > Barry was once proposing time-based releases; this idea didn't really > catch on. I'm still a proponent of this, but it doesn't seem like there's enough support for it. > It's the release manager who decides when the next bug fix release will > be made, often in response to developers asking for one. I'm happy to make a 2.6.5 release whenever it makes sense. -Barry From ziade.tarek at gmail.com Fri Jan 1 16:07:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:07:20 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04D81D51@TK5EX14MBXC118.redmond.corp.microsoft.com> <94bdd2610912141458m52da21ear4970506a37214e6@mail.gmail.com> <4dab5f760912230700l38a597f7l855bb2304025d6d5@mail.gmail.com> Message-ID: <94bdd2611001010707h4043ff64x7476049e435852b@mail.gmail.com> 2009/12/23 Glyph Lefkowitz : [..] > 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". > > The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. ?In other words it's got "bin", "lib", "share", "etc", etc.. ?~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation. Right, good idea Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Fri Jan 1 16:32:27 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:32:27 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> Message-ID: <94bdd2611001010732o38a563bcu6d0cc9122ed5c88f@mail.gmail.com> On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziad? wrote: > Hello, > > A while ago I've proposed to refactor the APIs that provides access to > the installation paths and configuration variables at runtime into a > single module called "sysconfig", and make it easier for all > implementations to work with them. I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek From status at bugs.python.org Fri Jan 1 18:07:11 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Jan 2010 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100101170711.A5AAF78654@psf.upfronthosting.co.za> ACTIVITY SUMMARY (12/25/09 - 01/01/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2537 open (+22) / 16902 closed (+19) / 19439 total (+41) Open issues with patches: 1016 Average duration of open issues: 705 days. Median duration of open issues: 462 days. Open Issues Breakdown open 2504 (+22) pending 32 ( +0) Issues Created Or Reopened (42) _______________________________ doc: Code is not always colored 12/30/09 CLOSED http://bugs.python.org/issue7487 reopened flox documention buglet for PyBuffer 12/26/09 CLOSED http://bugs.python.org/issue7577 created ronaldoussoren easy Behavior of operations on a closed file object is not documented 12/26/09 http://bugs.python.org/issue7578 created blep Patch to add docstrings to msvcrt 12/26/09 http://bugs.python.org/issue7579 created brian.curtin patch distutils makefile from python.org installer doesn't work on OS 12/27/09 http://bugs.python.org/issue7580 created bskaplan incomplete doc of zlib 12/27/09 CLOSED http://bugs.python.org/issue7581 created coolwanglu [patch] diff.py to use iso timestamp 12/27/09 http://bugs.python.org/issue7582 created techtonik patch doctest should normalize tabs when comparing output 12/27/09 http://bugs.python.org/issue7583 created techtonik datetime.rfcformat() for Date and Time on the Internet 12/27/09 http://bugs.python.org/issue7584 created techtonik [patch] difflib should separate filename from timestamp with tab 12/27/09 http://bugs.python.org/issue7585 created techtonik patch Typo in collections documentation 12/28/09 CLOSED http://bugs.python.org/issue7586 created nneonneo Python 3.1.1 source build issues 12/28/09 CLOSED http://bugs.python.org/issue7587 created sharmaag unittest.TestCase.shortDescription isn't short anymore 12/28/09 http://bugs.python.org/issue7588 created exarkun setup.py shouldn't try to build nis module when nis headers aren 12/28/09 CLOSED http://bugs.python.org/issue7589 created Arfrever patch 'exceptions' module mentioned in documentation 12/28/09 CLOSED http://bugs.python.org/issue7590 created July test_get_platform fails on 3.1 12/28/09 http://bugs.python.org/issue7591 created pitrou buildbot ssl module documentation: SSLSocket.unwrap description shown twi 12/28/09 http://bugs.python.org/issue7592 created mnewman Computed-goto patch for RE engine 12/29/09 http://bugs.python.org/issue7593 created akuchling patch, needs review shlex refactoring 12/29/09 http://bugs.python.org/issue7594 created ferringb patch, needs review Doc typo for select.kevent() 12/29/09 CLOSED http://bugs.python.org/issue7595 created whit537 test_logging sometimes fails 12/29/09 CLOSED http://bugs.python.org/issue7596 created pitrou curses.use_env implementation error 12/30/09 http://bugs.python.org/issue7597 created kanru patch Cannot install, problems with assembly? 12/30/09 CLOSED http://bugs.python.org/issue7598 created sh.00 (c)ElementTree can produce invalid XML 12/30/09 http://bugs.python.org/issue7599 created jwilk interpreter crashes before start 12/30/09 CLOSED http://bugs.python.org/issue7600 created mhh91 Installation - 2 tests failed: test_cmd_line test_xmlrpc 12/30/09 CLOSED http://bugs.python.org/issue7601 created sadhak Doc: make clean and make update do not delete or update Doc/tool 12/30/09 CLOSED http://bugs.python.org/issue7602 created flox There is no 'seq' type 12/30/09 CLOSED http://bugs.python.org/issue7603 created tom66 delattr __slots__ inconsistancy 12/30/09 CLOSED http://bugs.python.org/issue7604 created ferringb test_cmd_line fails with non-ascii path 12/30/09 http://bugs.python.org/issue7605 created pitrou test_xmlrpc fails with non-ascii path 12/30/09 http://bugs.python.org/issue7606 created pitrou stringlib fastsearch could be improved on 64-bit builds 12/30/09 http://bugs.python.org/issue7607 created pitrou PyUnicode_FromFormatV handles %R and %S incorrectly. 12/30/09 CLOSED http://bugs.python.org/issue7608 created alexandre.vassalotti Add --with-system-expat option 12/31/09 CLOSED http://bugs.python.org/issue7609 created Arfrever patch Cannot use both read and readline method in same ZipExtFile obje 12/31/09 http://bugs.python.org/issue7610 created lucifer shlex not posix compliant when parsing "foo#bar" 12/31/09 http://bugs.python.org/issue7611 created jjdmol2 Fix "pass and object" typo in Library Reference / Built-in Types 12/31/09 CLOSED http://bugs.python.org/issue7612 created ygale patch [cppcheck] found a regression : Invalid number of character ((). 12/31/09 CLOSED http://bugs.python.org/issue7613 created ettl.martin patch Python 2.6.4 segfaults 12/31/09 CLOSED http://bugs.python.org/issue7614 created ttsiod unicode_escape codec does not escape quotes 01/01/10 http://bugs.python.org/issue7615 created rhansen patch test_memoryview test_setitem_writable failures with Intel ICC 01/01/10 http://bugs.python.org/issue7616 created ivank distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option 01/01/10 http://bugs.python.org/issue7617 created Arfrever patch Issues Now Closed (46) ______________________ True division of integers could be more accurate 715 days http://bugs.python.org/issue1811 mark.dickinson patch API to clear most free lists 690 days http://bugs.python.org/issue2019 amaury.forgeotdarc patch unit test UnicodeWarning 687 days http://bugs.python.org/issue2100 ezio.melotti test_disutils fails 568 days http://bugs.python.org/issue3054 ronaldoussoren patch test_formatdate_usegmt fails on non-englisch locale 451 days http://bugs.python.org/issue4046 r.david.murray "??" in u"foo" raises a misleading error 410 days http://bugs.python.org/issue4328 r.david.murray zipfile - add __exit__ attribute to make ZipFile object compatib 287 days http://bugs.python.org/issue5511 ezio.melotti patch test_urllib2 fails - urlopen error file not on local host 271 days http://bugs.python.org/issue5625 orsenthil Improve --with-dbmliborder option 170 days http://bugs.python.org/issue6491 benjamin.peterson patch Move the special-case for integer objects out of PyBytes_FromObj 141 days http://bugs.python.org/issue6687 alexandre.vassalotti patch, 26backport setup.py fails to find headers of system libffi 104 days http://bugs.python.org/issue6943 benjamin.peterson patch, easy The replacement suggested for callable(x) in py3k is not equival 92 days http://bugs.python.org/issue7006 benjamin.peterson patch C/API - Document exceptions 89 days http://bugs.python.org/issue7033 lekma patch subprocess.check_output: "docstring has inconsistent leading whi 35 days http://bugs.python.org/issue7381 georg.brandl patch optparse Documentation References Example Files that do not Exis 30 days http://bugs.python.org/issue7404 georg.brandl patch datetime.datetime.isoformat truncation problem 29 days http://bugs.python.org/issue7413 amaury.forgeotdarc patch, needs review doc: Code is not always colored 0 days http://bugs.python.org/issue7487 georg.brandl float('nan')**2 != nan. float('nan')**2 error 33 on windows 13 days http://bugs.python.org/issue7534 mark.dickinson patch If a generator raises TypeError when being unpacked, an unrelate 10 days http://bugs.python.org/issue7548 amaury.forgeotdarc ctypes doc improvement: c_char_p 6 days http://bugs.python.org/issue7569 georg.brandl Strange issue : cursor.commit() with sqlite 5 days http://bugs.python.org/issue7572 ghaering tes_math fails Mac OS X 10.4 due to OverflowError in test_mtestf 5 days http://bugs.python.org/issue7575 mark.dickinson patch documention buglet for PyBuffer 2 days http://bugs.python.org/issue7577 georg.brandl easy incomplete doc of zlib 0 days http://bugs.python.org/issue7581 amaury.forgeotdarc Typo in collections documentation 0 days http://bugs.python.org/issue7586 georg.brandl Python 3.1.1 source build issues 0 days http://bugs.python.org/issue7587 amaury.forgeotdarc setup.py shouldn't try to build nis module when nis headers aren 1 days http://bugs.python.org/issue7589 benjamin.peterson patch 'exceptions' module mentioned in documentation 1 days http://bugs.python.org/issue7590 georg.brandl Doc typo for select.kevent() 0 days http://bugs.python.org/issue7595 georg.brandl test_logging sometimes fails 1 days http://bugs.python.org/issue7596 ezio.melotti Cannot install, problems with assembly? 0 days http://bugs.python.org/issue7598 ezio.melotti interpreter crashes before start 0 days http://bugs.python.org/issue7600 r.david.murray Installation - 2 tests failed: test_cmd_line test_xmlrpc 1 days http://bugs.python.org/issue7601 ezio.melotti Doc: make clean and make update do not delete or update Doc/tool 0 days http://bugs.python.org/issue7602 georg.brandl There is no 'seq' type 0 days http://bugs.python.org/issue7603 benjamin.peterson delattr __slots__ inconsistancy 0 days http://bugs.python.org/issue7604 benjamin.peterson PyUnicode_FromFormatV handles %R and %S incorrectly. 0 days http://bugs.python.org/issue7608 alexandre.vassalotti Add --with-system-expat option 1 days http://bugs.python.org/issue7609 benjamin.peterson patch Fix "pass and object" typo in Library Reference / Built-in Types 0 days http://bugs.python.org/issue7612 ezio.melotti patch [cppcheck] found a regression : Invalid number of character ((). 0 days http://bugs.python.org/issue7613 ezio.melotti patch Python 2.6.4 segfaults 0 days http://bugs.python.org/issue7614 mark.dickinson Fwd: Debian Bug#42318: urllib.py has problems with malformed pro 3436 days http://bugs.python.org/issue210849 shinnosuke urllib / urllib2 should cache 301 redirections 2425 days http://bugs.python.org/issue735515 pitrou fast modular exponentiation 2084 days http://bugs.python.org/issue936813 mark.dickinson patch bdist_deb - Debian packager 316 days http://bugs.python.org/issue1054967 astraw patch Carbon.Scrap.PutScrapFlavor 987 days http://bugs.python.org/issue1700507 ronaldoussoren Top Issues Most Discussed (10) ______________________________ 11 Add Option to Bind to a Local IP Address in httplib.py 462 days open http://bugs.python.org/issue3972 8 fast modular exponentiation 2084 days closed http://bugs.python.org/issue936813 6 [patch] difflib should separate filename from timestamp with ta 5 days open http://bugs.python.org/issue7585 6 [patch] diff.py to use iso timestamp 5 days open http://bugs.python.org/issue7582 6 Implement fastsearch algorithm for rfind/rindex 24 days open http://bugs.python.org/issue7462 5 test_xmlrpc fails with non-ascii path 2 days open http://bugs.python.org/issue7606 5 test_logging sometimes fails 1 days closed http://bugs.python.org/issue7596 5 Patch to add docstrings to msvcrt 6 days open http://bugs.python.org/issue7579 5 Distutils-based installer does not detect 64bit versions of Pyt 126 days open http://bugs.python.org/issue6792 5 _sha256 et al. encode to UTF-8 by default 17 days open http://bugs.python.org/issue3745 From stefan_ml at behnel.de Sun Jan 3 09:11:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jan 2010 09:11:16 +0100 Subject: [Python-Dev] Providing support files to assist 3.x extension authors In-Reply-To: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> References: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> Message-ID: Case Vanhorsen, 20.12.2009 01:38: > When I ported gmpy (Python to GMP multiple precision library) to > Python 3.x, I began to use PyLong_AsLongAndOverflow frequently. I > found the code to slightly faster and cleaner than using PyLong_AsLong > and checking for overflow. You might want to look at the code Cython generates for integer type conversions. We use specialised coercion code that gets generated on-the-fly to convert Python long/int from and to all sorts of C integer types with compile time (portability) and runtime size/value checks. Depending on your needs, this may or may not be faster than the above C-API function. Stefan From david.lyon at preisshare.net Mon Jan 4 00:42:15 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 10:42:15 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> Message-ID: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> > On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: > > This new operator removes the ambiguity the original proposal had, > without making it more > complex for common use cases. So if you dislike it, you will need to > propose something > else that also fixes the ambiguity we had. Ok. > Environment markers >.. >Here are some example of fields using such markers: > >Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: [Windows] pywin32 1.0+ That's simpler, shorter, and less ambiguous. Easier to parse for package managers. David From python at mrabarnett.plus.com Mon Jan 4 01:14:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 04 Jan 2010 00:14:43 +0000 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4132F3.7020905@mrabarnett.plus.com> David Lyon wrote: >> On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: >> >> This new operator removes the ambiguity the original proposal had, >> without making it more >> complex for common use cases. So if you dislike it, you will need to >> propose something >> else that also fixes the ambiguity we had. > > Ok. > >> Environment markers >> .. >> Here are some example of fields using such markers: >> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. > 'win32' is more specific than 'Windows' and, to me, '1.0+' means '>=1.0', not '>1.0'. From martin at v.loewis.de Mon Jan 4 01:21:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jan 2010 01:21:53 +0100 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4134A1.5090203@v.loewis.de> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. Don't you want the PEP to complete? Why this bike-shedding? I can agree it's shorter. I can't agree that it's simpler, or less ambiguous. Regards, Martin From ssteinerx at gmail.com Mon Jan 4 01:29:14 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 3 Jan 2010 19:29:14 -0500 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: On Jan 3, 2010, at 7:21 PM, Martin v. L?wis wrote: >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Really.... Enough, already! S From david.lyon at preisshare.net Mon Jan 4 01:47:56 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 11:47:56 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: <1349.218.214.45.58.1262566076.squirrel@syd-srv02.ezyreg.com> Hi Martin, Happy New Year, >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Well, I'm just helping out by pointing out some simpler ways as Tarek asked me. I was only answering his question. I was out celebrating so it took longer to reply than normal. Bike-Shedding ? Me ? which bikeshed? wanting simple? Anyway, I'm just reading the PEPs and commenting. If there are some alterations that can be done, lets discuss them. > I can agree it's shorter. > .. Cool. What I'd really like is a 'Code-Repository:' keyword so that we can install programs/libraries directly into a system. I feel that this would really simplify the packaging landscape, making it much easier for the scientific community and others to do python software installs. This would allow us to perphaps have something even *more modern* than CPAN. So yes, getting PEP-345 right is important to me. Have a nice day. David From abpillai at gmail.com Mon Jan 4 10:08:05 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 4 Jan 2010 14:38:05 +0530 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: <8548c5f31001040108v635a78ech33521371c6c50e82@mail.gmail.com> On Sun, Dec 27, 2009 at 11:21 PM, anatoly techtonik wrote: > On Tue, Dec 15, 2009 at 12:37 AM, Steven Bethard > wrote: > > > > If you're only concerned about 2.X, then yes, optparse will *never* be > > removed from 2.X. There will be a deprecation note in the 2.X > > documentation but deprecation warnings will only be issued when the -3 > > flag is specified. Please see the "Deprecation of optparse" section of > > the PEP: > > > > http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Ref http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse . Considering that optparse will be deprecated like 5 years from now. I think this point is moot. The deprecation strategy IMHO is perhaps way too conservative. Maybe it should be deprecated faster ;) -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Mon Jan 4 10:32:10 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 4 Jan 2010 10:32:10 +0100 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: >> If you're only concerned about 2.X, then yes, optparse will *never* be >> removed from 2.X. There will be a deprecation note in the 2.X >> documentation but deprecation warnings will only be issued when the -3 >> flag is specified. Please see the "Deprecation of optparse" section of >> the PEP: >> >> http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Compare: > http://docs.python.org/library/optparse.html > http://argparse.googlecode.com/svn/tags/r101/doc/index.html > > argparse should be recommended as advanced and more featured variant > of optparse - that goes without saying, but forcing people to switch > and annoying them with deprecation messages brings only headache. As has been noted already nobody is forcing people to switch. Optparse will be available as a separate package and everybody will be free to install it and will not have any deprecation messages anywhere. > Just > like optparse is better getopt, the latter could also be useful for > people coming from other languages and porting their libraries to > Python. > > I would better concentrate on real code examples how argparse solves > problems and would really want to see > http://www.python.org/dev/peps/pep-0389/#out-of-scope-various-feature-requests > implemented until argparse enters phase where backward incompatible > changes in API are impossible. > > I would prefer to see PEP 389 as a document describing proposed > solutions to argument parsing problems rather than petition to replace > one library with another. So, it should display common argument > parsing scenarios (use cases) with examples that are also useful > recipes for documentation. I guess more than 90% people here doesn't > have time to read argparse methods descriptions to see what they could > be useful for. -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From olemis at gmail.com Mon Jan 4 15:24:05 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 4 Jan 2010 09:24:05 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> Message-ID: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: > Thanks for the quick response. > > On 30/12/2009, Benjamin Peterson wrote: >> >> When I made that change, I didn't know that the __unittest "hack" was >> being used elsewhere outside of unittest, so I felt fine replacing it >> with another. While I still consider it an implementation detail, I >> would be ok with exposing an "official" API for this. Perhaps >> __unittest_ignore_traceback? > > Well, bazaar has had the trick for a couple of years, and googling > around now turns up some other projects using it or thinking about it: > > > > Add `dutest` and probably `nose` [2]_ and ... > Reinstating the old implementation (with the same name) would mean > that existing code would keep working with Python 2.7 IOW ... if it is removed all the libraries will have to change and check for the Py version and ... (probably not a big deal depending on the details of the ?solution?) > but maybe a > discussion could start about a new, less hacky, way of doing the same > I am strongly -1 for modifying the classes in ?traditional? unittest module [2]_ (except that I am strongly +1 for the package structure WITHOUT TOUCHING anything else ...) , and the more I think about it I am more convinced ... but anyway, this not a big deal (and in the end what I think is not that relevant either ... :o). So ... pass > May not be worthwhile making life more complicated though, > there aren't *that* many unittest-extending projects. > But every library extending `unittest` will do it or otherwise not-so-useful error messages will be displayed ;o) PS: Happy New Year ! BTW .. [1] [feature] nosexml should omit trace frames where `__unittest... (http://code.google.com/p/python-nosexml/issues/detail?id=4#c0) .. [2] Assessment of unittest 2.7 API : new features and opinions (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free new ripit 1.3.3 Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/KFwyUTKH0vI/ From juanfhj at gmail.com Tue Jan 5 17:10:16 2010 From: juanfhj at gmail.com (Juan Fernando Herrera J.) Date: Tue, 5 Jan 2010 11:10:16 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility Message-ID: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Jan 5 17:50:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 05 Jan 2010 16:50:15 +0000 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B436DC7.8040605@voidspace.org.uk> On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way > major software hasn't been ported to it. I'm eager to use 3, but > paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me > if this has been suggested in the past. > I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well. I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter). All the best, Michael Foord > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Jan 5 18:21:31 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 5 Jan 2010 11:21:31 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion. I'm sure everyone out there would like to dive into supporting 3.x, but it's tough to prioritize when you are up to your eyeballs with 2.x bugs in your tracker. Look at Twisted ( http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3) -- over 1000 issues, ~5 developers -- 3.x support won't be here tomorrow, but it's on the horizon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 18:23:08 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2010 09:23:08 -0800 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B436DC7.8040605@voidspace.org.uk> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B436DC7.8040605@voidspace.org.uk> Message-ID: On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord wrote: > On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > > I don't know about other developers, but I certainly expected Python 3 > adoption to take a few years due to libraries only gradually making the > jump. I also expected there to be nervousness and pain around the migration > as well. > > I think in fact that libraries *are* migrating and there are lots of > encouraging signs. As a community we need to do all we can to promote Python > 3 adoption, but I don't think there is much reason to be worried (or > complacent for that matter). > Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Jan 5 18:52:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 18:52:20 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <319e029f1001050952o71cb29afh66445550beeb99c2@mail.gmail.com> On Tue, Jan 5, 2010 at 17:10, Juan Fernando Herrera J. wrote: > I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. Yes. Python 3 is not what you want to use today if you write applications. If you write libraries 2010 is the year to port, IMO. With some luck, 2011 will be the year to start porting applications properly. We'll see. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ianb at colorstudy.com Tue Jan 5 20:00:49 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 13:00:49 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin wrote: > On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > >> How about a new python 3 release with (possibly partial) backwards >> compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major >> software hasn't been ported to it. I'm eager to use 3, but paradoxically, >> the 3 release makes me rather stuck with 2.6. Excuse me if this has been >> suggested in the past. >> >> > The proper route to take, in my opinion, is to see what 2.x libraries you > are using that are not 3.x compatible, run 2to3 on them, then run their test > suite, and see where you get. Submit a patch or two to the library and see > what happens -- it at least gets the wheels in motion. > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 5 21:24:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 5 Jan 2010 15:24:45 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> On Jan 5, 2010, at 2:00 PM, Ian Bicking wrote: > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. It seems like this is a problem to be addressed, then. Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. > The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). No-op constructions like 'bytes("")' could help for older versions of Python, though. A very, very small runtime shim could provide support for these, if 2to3 could be told about it somehow. > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? Sort of like our own version of the #line directive? :) Seriously though, I find it hard to believe that this is a big problem. The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. > I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. This is almost certainly more of a pain than trying to trick 2to3 into doing the right thing. From regebro at gmail.com Tue Jan 5 21:57:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 21:57:35 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <319e029f1001051257k3827c52ahcd115b15d9a8ece7@mail.gmail.com> On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz wrote: > It seems like this is a problem to be addressed, then. ?Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x. >> The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated. > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 5 22:07:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 22:07:23 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B43AA0B.5030308@v.loewis.de> > It's not even that easy -- libraries can't apply patches for Python 3 > compatibility as they usually break Python 2 compatibility. Potentially > libraries could apply patches that make a codebase 2to3 ready, but from > what I've seen that's more black magic than straight forward updating, > as such patches have to trick 2to3 producing the output that is desired. I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x. > The only workable workflow I've seen people propose for maintaining a > single codebase with compatibility across both 2 and 3 is to use such > tricks, with aliases to suppress some 2to3 updates when they are > inappropriate I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way. > Also, running 2to3 on installation is kind of annoying, as you get > source that isn't itself the canonical source, so to fix bugs you have > to look at the installed source and trace it back to the bug in the > original source. That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin From martin at v.loewis.de Tue Jan 5 22:13:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jan 2010 22:13:07 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <4B43AB63.3000607@v.loewis.de> > No-op constructions like 'bytes("")' could help for older versions of > Python, though. A very, very small runtime shim could provide > support for these, if 2to3 could be told about it somehow. You actually don't *need* 2to3 support for that - bytes("") can work in either version: 2.x: def bytes(s): return s 3.x: def bytes(s) return s.encode("ascii") On top of that, you *could* as for bytes("") to be converted to b"" in 3.x - and it is indeed possible to tell 2to3 about that, and you don't even need to modify 2to3's source to do so: it can be part of your package. >> Also, running 2to3 on installation is kind of annoying, as you get >> source that isn't itself the canonical source, so to fix bugs you >> have to look at the installed source and trace it back to the bug >> in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in > .pycs rather than based on where the code was actually loaded in the > filesystem, couldn't 2to3 could do .pyc rewriting to point at the > original source? Sort of like our own version of the #line > directive? :) I think it could, but it would be fairly flaky as the pycs can get lost, and lose that information after regeneration. Also, it may be confusing in other scenarios. I'd rather have it generate separate mapper files, and change the traceback printing to consider these (as an option). > Seriously though, I find it hard to believe that this is a big > problem. The 3.x source looks pretty similar to the 2.x source, and > it's good to look at both if you're dealing with a 3.x issue. That's my experience as well - the only challenge is that you can't cut-n-paste the source URL into an editor. Regards, Martin From ianb at colorstudy.com Tue Jan 5 22:39:21 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 15:39:21 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43AA0B.5030308@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: On Tue, Jan 5, 2010 at 3:07 PM, "Martin v. L?wis" wrote: > > > It's not even that easy -- libraries can't apply patches for Python 3 > > compatibility as they usually break Python 2 compatibility. ?Potentially > > libraries could apply patches that make a codebase 2to3 ready, but from > > what I've seen that's more black magic than straight forward updating, > > as such patches have to trick 2to3 producing the output that is desired. > > I wouldn't qualify it in that way. It may be necessary occasionally to > trick 2to3, but that's really a bug in 2to3 which you should report, so > that trickery is then a work-around for a bug - something that you may > have to do with other API, as well. > > The "black magic" is really more in the parts that 2to3 doesn't touch > at all (because they are inherently not syntactic); these are the > problem areas Guido refers to. The "black magic" then is to make the > same code work unmodified for both 2.x and 3.x. Just to clarify, the black magic I'm referring to is things like: try: ?? ?unicode_ = unicode except NameError: ?? ?unicode_ = str and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). ?If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. ?It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. ?For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str. ?That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself. Also replying to Glyph: > > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? ?Sort of like our own version of the #line directive? :) > > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun. So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits. Which is a terribly crotchety of me. Sorry. -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From martin at v.loewis.de Tue Jan 5 23:23:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 23:23:53 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: <4B43BBF9.2000302@v.loewis.de> > Just to clarify, the black magic I'm referring to is things like: > > try: > unicode_ = unicode > except NameError: > unicode_ = str > > and some other aliases like this that are unambiguous and which 2to3 > won't touch (if you write them correctly). If the porting guide noted > all these tricks (of which several have been developed, and I'm only > vaguely aware of a few) that would be helpful. It's not a lot of > tricks, but the tricks are not obvious and 2to3 gets the translation > wrong pretty often without them. For instance, when I say str in > Python 2 I often means bytes, unsurprisingly, but 2to3 translates both > str and unicode to str. No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3? > That *nothing* translates to bytes by default > (AFAIK) means that people must either be living in a bytes-free world > (which sure, lots of code does) or they are using tricks not included > in 2to3 itself. By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3). > Since 2to3 maintains line numbers yes, it wouldn't be that bad. But > then I don't currently develop any code that is "installed", I only > develop code that is directly from a source code checkout, and where > the checkout is put on the path. I guess I could have something that > automatically builds the code on every edit, and that's not > infeasible. It's just not fun. Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it. > So long as I have to support Python 2 > (which is like forever) then adding Python 3 only makes development > that much more complicated and much less fun, with no concrete > benefits. I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin From ziade.tarek at gmail.com Wed Jan 6 01:08:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 01:08:34 +0100 Subject: [Python-Dev] PEP 386 and PEP 345 Message-ID: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Hi, I think we've reached a consensus on those two PEPs. Although, there's one last point that was forgotten in the discussions : I've introduced "rc" in the pre-releases markers, so PEP 386 is compatible with Python's own version scheme. "rc" comes right after "c" in the sorting. It's slightly redundant with the "c" marker but I don't think this really matters as long as consumers know how to order them (a < b < c < rc). I have also stated that "c" is the preferred marker for third party projects, from PEP 386 point of view. Is there anything else I can do to make those two PEPs accepted ? Regards Tarek -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Wed Jan 6 01:26:34 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:26:34 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43BBF9.2000302@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> <4B43BBF9.2000302@v.loewis.de> Message-ID: <1322.218.214.45.58.1262737594.squirrel@syd-srv02.ezyreg.com> Hi Martin, > ... but in the long run, starting now will have saved you a lot of > pressure in the long run, and stop users from switching away from > your packages because of lack of Python 3 support. In a production situation it works the other way around. If there's an application that requires twisted (or whatever package) then most people would use the appropriate interpreter to match the library. Since you guys all did your jobs so well :-) doing so is painless. Because there is so much "comfort" with the existing situation it makes it an effort for people to move to a different lounge chair. Namely python 3. I'd suggest that moving the package set (pypi) to python 3 could be kicked along with the help of some automated tools. I don't know what tools you guys have got. But I am very sure that if code analysis was provided to package developers on python 3 (so they don't have to run it themselves), then it would be like an even bigger tv screen in a bigger lounge room and it would assist in drawing them over. David From david.lyon at preisshare.net Wed Jan 6 01:36:24 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:36:24 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <1337.218.214.45.58.1262738184.squirrel@syd-srv02.ezyreg.com> Hi Ian, > The only workable workflow I've seen people propose for maintaining a > single > codebase with compatibility across both 2 and 3 is to use such tricks, > with > aliases to suppress some 2to3 updates when they are inappropriate, so that > you can run 2to3 on install and have a single canonical Python 2 source. > Python 2.7 won't help much (even though it is trying) as the introduction > of non-ambiguous constructions like b"" aren't compatible with previous > versions of Python and so can't be used in many libraries (support at > least > back to Python 2.5 is the norm for most libraries, I think). > > Also, running 2to3 on installation is kind of annoying, as you get source > that isn't itself the canonical source, so to fix bugs you have to look at > the installed source and trace it back to the bug in the original source. > > I suspect a reasonable workflow might be possible with hg and maybe patch > queues, but I don't feel familiar enough with those tools to map that out. That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David From david.lyon at preisshare.net Wed Jan 6 03:01:22 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 13:01:22 +1100 (EST) Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: <1617.218.214.45.58.1262743282.squirrel@syd-srv02.ezyreg.com> > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? Tarek, Given that I helped out so much last year with the PEP in discussing different options, even if they weren't accepted, I really feel that it is unfair if my name isn't mentioned. It was a huge time sacrifice on my part. For example, even if I only managed to explain the version numbering and clarify how that worked. It did take me some time to do that. What I did do however, was spend a lot of time with the multiplatform "Markers". I still think that two short weeks more of "discussion" could resolve some issues. That discussion went for 4 months on distutils-ml. Look, major issues aside, can you make the following concessions on PEP-345 which I only feel will help it, namely: 1) Source-Repository: specify a code repository to install from 2) Streamline Requires-Python: by implementing "Markers" as noted by the PEP. A marker being something like "Requires-Python(windows): lxml". Otherwise remove the word marker from the PEP and just replace with "metacode". Markers are what were discussed on distutils-ml. Metacode is what is described in the PEP. 3) Remove the inconsistency and platform ambiguities. Especially for windows users. For example, "win32" is extremely confusing for windows users right now. As more and more systems now are 64 bit. Use the platform module, instead of the sys module for constants. I'll post to distutils-ml on this. I am certainly not trying to hold this PEP up, and I apologise on my part for my late attention. I will post to distutils-ml on these and i promise to keep my comments unheated and unwitty. Having said that, PEP-345 is *super-important*. A week or two or three more discussion and the issues can be resolved. We all just want to focus on being productive. It would be a great accomplishment for you to get PEP-345 approved and likewise for me getting mentioned even in a minor role as helping out on a PEP. So I'm hoping that you can make a few last minute concessions meaning that we can all happily go on our way in 2010. Regards David From brett at python.org Wed Jan 6 06:20:30 2010 From: brett at python.org (Brett Cannon) Date: Tue, 5 Jan 2010 21:20:30 -0800 Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 16:08, Tarek Ziad? wrote: > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? > As you said, consensus has been reached, so just Guido's BDFL stamp of approval is all I can think of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Jan 6 12:19:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:19:45 +0000 Subject: [Python-Dev] bug triage Message-ID: <4B4471D1.9020707@simplistix.co.uk> Hi All, Is there a high volume of incoming bugs to the Python tracker? If so, I'd like to help with triaging. I think I have all the necessary access, what I'm missing is the knowledge of how to set myself up to get notifications of new bugs... How do I do that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at voidspace.org.uk Wed Jan 6 12:24:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 06 Jan 2010 11:24:37 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: <4B4472F5.8000709@voidspace.org.uk> On 06/01/2010 11:19, Chris Withers wrote: > Hi All, > > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the > necessary access, what I'm missing is the knowledge of how to set > myself up to get notifications of new bugs... > > How do I do that? > Bug triaging is one of Python's "big needs" and anything you do to help on this score would be much appreciated. Particularly reviewing new and outstanding issues. I assumed there would be RSS feeds for bug tracker activity but can't easily find these on the tracker. There is a bot that posts activity to #python-dev, so there must be some way of getting this information. All the best, Michael > cheers, > > Chris > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Jan 6 12:25:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 11:25:16 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: Hi Chris, > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Do you really want to get such notifications? There may be a lot of them. If you want however, you can join #python-dev on IRC (irc.freenode.net) where there's a bot which posts updates of all bugs on the tracker. There's usually not a lot of discussion going on so you probably won't feel flooded. In addition to bug triage, what is needed is reviewing of existing patches, as well as writing patches for issues which haven't been addressed yet :-) Regards Antoine. From chris at simplistix.co.uk Wed Jan 6 12:30:40 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:30:40 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B447460.7080100@simplistix.co.uk> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. Yeah, email-out is what I'm really after... I have it for my own Roundup instance so it can't be that hard to do ;-) Roch?, you guys host the bug tracker, right? Is there email-out set up for it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Jan 6 12:37:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 21:37:23 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B4475F3.5040406@gmail.com> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. I'm pretty sure the bugs list is still the primary spooled notification mechanism: http://mail.python.org/mailman/listinfo/python-bugs-list There are also the weekly tracker activity summaries that are posted here to python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From chris at simplistix.co.uk Wed Jan 6 12:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:41:28 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <4B4476E8.8050709@simplistix.co.uk> Nick Coghlan wrote: > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list That's what I was after, thanks! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From facundobatista at gmail.com Wed Jan 6 13:03:08 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 6 Jan 2010 09:03:08 -0300 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: On Wed, Jan 6, 2010 at 8:19 AM, Chris Withers wrote: > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Not notifications, but maybe a way to have a higher look of them for easy selection: http://www.taniquetil.com.ar/cgi-bin/pytickets.py Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From ziade.tarek at gmail.com Wed Jan 6 13:29:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:29:58 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord wrote: > On 06/01/2010 11:19, Chris Withers wrote: >> >> Hi All, >> >> Is there a high volume of incoming bugs to the Python tracker? >> If so, I'd like to help with triaging. I think I have all the necessary >> access, what I'm missing is the knowledge of how to set myself up to get >> notifications of new bugs... >> >> How do I do that? >> > > Bug triaging is one of Python's "big needs" and anything you do to help on > this score would be much appreciated. Particularly reviewing new and > outstanding issues. Another useful triage I think, is to review the oldest bugs (some of them are > 5 years) and remove the ones that are not relevant anymore, or duplicate with newer entries. Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Wed Jan 6 13:31:08 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 12:31:08 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B44828C.4070201@simplistix.co.uk> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with someone as a paired task for those 2 days... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Jan 6 13:37:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:37:55 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B44828C.4070201@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B44828C.4070201@simplistix.co.uk> Message-ID: <94bdd2611001060437s2d0242aembc669c3263773fea@mail.gmail.com> On Wed, Jan 6, 2010 at 1:31 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Another useful triage I think, is to review the oldest bugs (some of >> them are > 5 years) >> and remove the ones that are not relevant anymore, or duplicate with >> newer entries. > > I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with > someone as a paired task for those 2 days... I'll be doing Distutils stuff but I can probably help around a bit in that task: Distutils have quite a few old issues so I can tackle those From roche at upfrontsystems.co.za Wed Jan 6 13:32:39 2010 From: roche at upfrontsystems.co.za (=?ISO-8859-1?Q?Roch=E9?= Compaan) Date: Wed, 06 Jan 2010 14:32:39 +0200 Subject: [Python-Dev] bug triage In-Reply-To: <4B447460.7080100@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B447460.7080100@simplistix.co.uk> Message-ID: <1262781159.2836.218.camel@didi> On Wed, 2010-01-06 at 11:30 +0000, Chris Withers wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > Yeah, email-out is what I'm really after... I have it for my own Roundup > instance so it can't be that hard to do ;-) > > Roch?, you guys host the bug tracker, right? Is there email-out set up > for it? We do, but we don't administer it. There are a few administrators taking care of it and you should be able to reach them by logging your request here: http://psf.upfronthosting.co.za/roundup/meta/ or post it to the Infrastructure mailing list: infrastructure at python.org -- Roch? Compaan Upfront Systems http://www.upfrontsystems.co.za From ncoghlan at gmail.com Wed Jan 6 13:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 22:57:24 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B4488B4.2070208@gmail.com> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I believe someone (Daniel Diniz, maybe?) did do a pass over those some time in the last 12 months, so most of the obviously irrelevant ones that are that old should already be gone. Not to say it isn't worth doing another pass, just saying not to get disheartened if there aren't many that can be readily closed. There are at least a few still kicking around just because they're difficult to deal with (there's an ancient one to do with one of the ways circular imports can fail that I occasionally go back and reread before moving on to something more tractable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Wed Jan 6 14:43:17 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 08:43:17 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <4B4476E8.8050709@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> <4B4476E8.8050709@simplistix.co.uk> Message-ID: <20100106134317.3EF141FB5A6@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:41:28 +0000, Chris Withers wrote: > Nick Coghlan wrote: > > I'm pretty sure the bugs list is still the primary spooled notification > > mechanism: > > http://mail.python.org/mailman/listinfo/python-bugs-list > > That's what I was after, thanks! Just for completeness, there's also new-bugs-announce if you want just *new* bug notification. That's more for people who want to watch for bugs they want to become nosy on, though; if you are doing triage python-bugs-list is what you want. Please also read http://www.python.org/dev/workflow/ if you haven't already. Thanks for being willing to chip in! --David From brian.curtin at gmail.com Wed Jan 6 15:57:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 08:57:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4488B4.2070208@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > I believe someone (Daniel Diniz, maybe?) did do a pass over those some > time in the last 12 months, so most of the obviously irrelevant ones > that are that old should already be gone. Not to say it isn't worth > doing another pass, just saying not to get disheartened if there aren't > many that can be readily closed. > > There are at least a few still kicking around just because they're > difficult to deal with (there's an ancient one to do with one of the > ways circular imports can fail that I occasionally go back and reread > before moving on to something more tractable). > > Cheers, > Nick. > On the topic of bugs that can be readily closed (literally), I've recently come across a number of issues which appear to be sitting in a patch or review stage, but their patches have been committed and the issue remains open. What is the best course of action there? I'd just go ahead and close the issue myself but I don't have tracker privileges. I'm willing to help out with another Daniel Diniz-esque triage sweep if that would help. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Wed Jan 6 16:14:20 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 6 Jan 2010 10:14:20 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <7FCF8B19-3465-4392-80CC-BEAEBD926ECA@gmail.com> On Jan 6, 2010, at 7:29 AM, Tarek Ziad? wrote: > On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord > wrote: >> On 06/01/2010 11:19, Chris Withers wrote: >>> >>> Hi All, >>> >>> Is there a high volume of incoming bugs to the Python tracker? >>> If so, I'd like to help with triaging. I think I have all the necessary >>> access, what I'm missing is the knowledge of how to set myself up to get >>> notifications of new bugs... >>> >>> How do I do that? >>> >> >> Bug triaging is one of Python's "big needs" and anything you do to help on >> this score would be much appreciated. Particularly reviewing new and >> outstanding issues. > > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I was actually thinking about that the other day when I saw that the average age of bugs on the Python tracker was at some hideously large 3 digit number. The 'success' statistic would be to bring that down below, say, 100. S From skip at pobox.com Wed Jan 6 17:38:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 6 Jan 2010 10:38:13 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <19268.48245.616046.874126@montanaro.dyndns.org> >>>>> "Nick" == Nick Coghlan writes: Nick> I'm pretty sure the bugs list is still the primary spooled Nick> notification mechanism: Nick> http://mail.python.org/mailman/listinfo/python-bugs-list Actually, there is a new-bugs-announce list: http://mail.python.org/mailman/listinfo/new-bugs-announce Skip From solipsis at pitrou.net Wed Jan 6 18:56:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 17:56:49 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > On the topic of bugs that can be readily closed (literally), I've > recently come across a number of issues which appear to be sitting in a > patch or review stage, but their patches have been committed and the > issue remains open. What is the best course of action there? Post a message on the issue asking for info. From solipsis at pitrou.net Wed Jan 6 19:09:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 18:09:39 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > > On the topic of bugs that can be readily closed (literally), I've > > recently come across a number of issues which appear to be sitting in a > > patch or review stage, but their patches have been committed and the > > issue remains open. What is the best course of action there? > > Post a message on the issue asking for info. Ok, I realize my answer might have been a bit terse :-) The patch might be waiting to be merged in all development branches, or it may not totally resolve the issue, or perhaps documentation needs to be updated, or perhaps it is pending a verdict from the buildbots, etc. You can't deduce that the issue is completely fixed from the simple fact that something has been committed. Regards Antoine Pitrou. From brett at python.org Wed Jan 6 20:03:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 11:03:32 -0800 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > >> I believe someone (Daniel Diniz, maybe?) did do a pass over those some >> time in the last 12 months, so most of the obviously irrelevant ones >> that are that old should already be gone. Not to say it isn't worth >> doing another pass, just saying not to get disheartened if there aren't >> many that can be readily closed. >> >> There are at least a few still kicking around just because they're >> difficult to deal with (there's an ancient one to do with one of the >> ways circular imports can fail that I occasionally go back and reread >> before moving on to something more tractable). >> >> Cheers, >> Nick. >> > > On the topic of bugs that can be readily closed (literally), I've recently > come across a number of issues which appear to be sitting in a patch or > review stage, but their patches have been committed and the issue remains > open. What is the best course of action there? I'd just go ahead and close > the issue myself but I don't have tracker privileges. > > If a core developer is willing to step forward and vouch for you to get tracker privileges then I will give them to you. We are trying to give out tracker privs w/ less time than required to get commit privileges. So as long as you have helped out on a few issues in a positive and correct way that should be enough to get one of the regulars who perform triage to notice. -Brett I'm willing to help out with another Daniel Diniz-esque triage sweep if that > would help. > > Brian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Wed Jan 6 21:41:05 2010 From: nad at acm.org (Ned Deily) Date: Wed, 06 Jan 2010 12:41:05 -0800 Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: In article <4B4475F3.5040406 at gmail.com>, Nick Coghlan wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list Also, that mailing list (along with most python development related mailing lists) is mirrored at gmane.org which means it can also be obtained via a newsreader (NNTP) or various RSS feeds. http://dir.gmane.org/gmane.comp.python.bugs -- Ned Deily, nad at acm.org From nick.bastin at gmail.com Wed Jan 6 22:14:54 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:14:54 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? Message-ID: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> (This may occur on more platforms - I can test on more unix platforms if the consensus is this is an actual problem and I'm not just a nut) On freebsd5, if you do a simple ./configure --enable-shared in current (2.7) trunk, your python shared library will build properly, but all modules will fail to find the shared library and thus fail to build: gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.7/u1/Python/Python-2.7a1/Modules/_struct.o -L/u1/tmp/python2.7a1/lib -L/usr/local/lib -lpython2.7 -o build/lib.freebsd-5.3-RELEASE-i386-2.7/_struct.so /usr/bin/ld: cannot find -lpython2.7 building '_ctypes_test' extension ... This of course is because libpython2.7.so is in the current directory and not (yet) installed in /usr/local/lib. I've made a very simple fix for this problem that works, but at least to me smells a bit funny, which is to modify setup.py to add the following to detect_modules(): # If we did --enable-shared, we need to be able to find the library # we just built in order to build the modules. if platform == 'freebsd5': add_dir_to_list(self.compiler_obj.library_dirs, '.') Which brings me to a few questions: a) Does this seem like a real problem, or am I missing something obvious? b) Does this fix seem like the sensible thing to do? (it seems at least that we ought to check that the user configured --enable-shared and only set -L. in that case, if that's possible) Setting --enable-shared when you actually have a libpython2.7.so in /usr/local/lib (or whatever --prefix you've selected) is possibly even more dangerous, because it may succeed in linking against a differently-built library than what you intended. -- Nick From nick.bastin at gmail.com Wed Jan 6 22:39:17 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:39:17 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <66d0a6e11001061339t38202b70mca1091e1926ecf4b@mail.gmail.com> On Wed, Jan 6, 2010 at 16:14, Nicholas Bastin wrote: > This of course is because libpython2.7.so is in the current directory > and not (yet) installed in /usr/local/lib. One minor correction - as you could see from the compile line, the actual --prefix in this case is /u1/tmp/python2.7a1, but the libraries obviously aren't installed there yet either. Perhaps a better fix than setting -L. would be to put the shared library in build/lib.freebsd-5.3-RELEASE-i386-2.7 and add that to the library path for the linker (the build creates this directory, but installs nothing in it). -- Nick From martin at v.loewis.de Wed Jan 6 23:21:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jan 2010 23:21:44 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <4B450CF8.8090009@v.loewis.de> > b) Does this fix seem like the sensible thing to do? No. Linking in setup.py should use the same options as if the module was built as *shared* through Modules/Setup, which, IIUC, should use BLDLIBRARY. Regards, Martin From nick.bastin at gmail.com Thu Jan 7 01:08:13 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 19:08:13 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B450CF8.8090009@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> Message-ID: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> On Wed, Jan 6, 2010 at 17:21, "Martin v. L?wis" wrote: >> b) Does this fix seem like the sensible thing to do? > > No. Linking in setup.py should use the same options as if the module > was built as *shared* through Modules/Setup, which, IIUC, should use > BLDLIBRARY. Thanks for that pointer, that makes much more sense. Indeed, BLDLIBRARY on FreeBSD* is set to '-L. -lpython$(VERSION)' if you set --enable-shared, but somehow that piece of information doesn't propagate into the module build. More investigation to be done... -- Nick From rdmurray at bitdance.com Thu Jan 7 02:22:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 20:22:42 -0500 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > > On the topic of bugs that can be readily closed (literally), I've recently > > come across a number of issues which appear to be sitting in a patch or > > review stage, but their patches have been committed and the issue remains > > open. What is the best course of action there? I'd just go ahead and close > > the issue myself but I don't have tracker privileges. > > > > > If a core developer is willing to step forward and vouch for you to get > tracker privileges then I will give them to you. We are trying to give out > tracker privs w/ less time than required to get commit privileges. So as > long as you have helped out on a few issues in a positive and correct way > that should be enough to get one of the regulars who perform triage to > notice. > > -Brett I've done a quick scan of issues Brian is nosy on to refresh my memory, and I'd say he's definitely been making positive contributions. I'm willing to volunteer to keep an eye on his triage work for a while if you grant him tracker privs. Brian, I assume you'll be cognizant of Antoine's advice about making sure a bug really should be closed before closing it :) Hanging out in #python-dev on freenode while working on issues can be helpful, as well, since you can quickly ask whoever is there for second opinions on particular bugs. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Thu Jan 7 02:28:26 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 17:28:26 -0800 Subject: [Python-Dev] bug triage In-Reply-To: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > > On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > > On Wed, Jan 6, 2010 at 06:57, Brian Curtin > wrote: > > > On the topic of bugs that can be readily closed (literally), I've > recently > > > come across a number of issues which appear to be sitting in a patch or > > > review stage, but their patches have been committed and the issue > remains > > > open. What is the best course of action there? I'd just go ahead and > close > > > the issue myself but I don't have tracker privileges. > > > > > > > > If a core developer is willing to step forward and vouch for you to get > > tracker privileges then I will give them to you. We are trying to give > out > > tracker privs w/ less time than required to get commit privileges. So as > > long as you have helped out on a few issues in a positive and correct way > > that should be enough to get one of the regulars who perform triage to > > notice. > > > > -Brett > > I've done a quick scan of issues Brian is nosy on to refresh my > memory, and I'd say he's definitely been making positive contributions. > I'm willing to volunteer to keep an eye on his triage work for a while > if you grant him tracker privs. > > Done for the username brian.curtin (email doesn't match the one Brian emailed from so do let me know, Brian if this is the right username). Welcome aboard! -Brett > Brian, I assume you'll be cognizant of Antoine's advice about making > sure a bug really should be closed before closing it :) Hanging out in > #python-dev on freenode while working on issues can be helpful, as well, > since you can quickly ask whoever is there for second opinions on > particular bugs. > > -- > R. David Murray www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Jan 7 02:59:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 19:59:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 19:28, Brett Cannon wrote: > > > On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > >> >> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: >> > On Wed, Jan 6, 2010 at 06:57, Brian Curtin >> wrote: >> > > On the topic of bugs that can be readily closed (literally), I've >> recently >> > > come across a number of issues which appear to be sitting in a patch >> or >> > > review stage, but their patches have been committed and the issue >> remains >> > > open. What is the best course of action there? I'd just go ahead and >> close >> > > the issue myself but I don't have tracker privileges. >> > > >> > > >> > If a core developer is willing to step forward and vouch for you to get >> > tracker privileges then I will give them to you. We are trying to give >> out >> > tracker privs w/ less time than required to get commit privileges. So as >> > long as you have helped out on a few issues in a positive and correct >> way >> > that should be enough to get one of the regulars who perform triage to >> > notice. >> > >> > -Brett >> >> I've done a quick scan of issues Brian is nosy on to refresh my >> memory, and I'd say he's definitely been making positive contributions. >> I'm willing to volunteer to keep an eye on his triage work for a while >> if you grant him tracker privs. >> >> > Done for the username brian.curtin (email doesn't match the one Brian > emailed from so do let me know, Brian if this is the right username). > Welcome aboard! > > Yep, that's the one. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Jan 7 04:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 07 Jan 2010 03:07:56 +0000 Subject: [Python-Dev] GIL required for _all_ Python calls? Message-ID: <4B45500C.8090905@mrabarnett.plus.com> Hi, I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? Or is it just a case of checking the source files for mentions of the GIL? The header file for PyList_New, for example, doesn't mention it! Thanks From john.arbash.meinel at gmail.com Thu Jan 7 04:25:48 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 06 Jan 2010 21:25:48 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B45543C.2090200@gmail.com> MRAB wrote: > Hi, > > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? Or is it just a case of checking the source > files for mentions of the GIL? The header file for PyList_New, for > example, doesn't mention it! > > Thanks Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may get concurrent updating of the value, and then the final value is wrong. (two threads do 5+1 getting 6, rather than 7, and when the decref, you end up at 4 rather than back at 5). AFAIK, the only things that don't require the GIL are macro functions, like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for example, will be increfing and setting the exception state, so certainly needs the GIL to be held. John =:-> From benjamin at python.org Thu Jan 7 04:32:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jan 2010 21:32:17 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45543C.2090200@gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> Message-ID: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> 2010/1/6 John Arbash Meinel : > Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may > get concurrent updating of the value, and then the final value is wrong. > (two threads do 5+1 getting 6, rather than 7, and when the decref, you > end up at 4 rather than back at 5). Correct. > > AFAIK, the only things that don't require the GIL are macro functions, > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > example, will be increfing and setting the exception state, so certainly > needs the GIL to be held. As a general rule, I would say, no Py* macros are safe without the gil either (the exception being Py_END_ALLOW_THREADS), since they mutate Python objects which must be protected. -- Regards, Benjamin From guido at python.org Thu Jan 7 05:29:03 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Jan 2010 20:29:03 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: On Wed, Jan 6, 2010 at 7:32 PM, Benjamin Peterson wrote: > 2010/1/6 John Arbash Meinel : > > AFAIK, the only things that don't require the GIL are macro functions, > > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > > example, will be increfing and setting the exception state, so certainly > > needs the GIL to be held. > > As a general rule, I would say, no Py* macros are safe without the gil > either (the exception being Py_END_ALLOW_THREADS), since they mutate > Python objects which must be protected. That's keeping it on the safe side, since there are some macros like PyString_AS_STRING() that are also safe, *if* you are owning at least one reference to the string object. At the same time, "no Py* macros" is not quite strong enough, since if you called PyString_AS_STRING() before releasing the GIL but you don't own a reference to the string object, the string might be deallocated behind your back by another thread. A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). -- --Guido van Rossum (python.org/~guido) From yoann.padioleau at facebook.com Thu Jan 7 08:24:42 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Wed, 6 Jan 2010 23:24:42 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt Message-ID: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a file called ast.txt. Where does this file come from ? Is it generated from Python.asdl ? From johan.gill at agama.tv Thu Jan 7 10:46:52 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 10:46:52 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. Message-ID: <4B45AD8C.5000405@agama.tv> Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before, so this is probably an already solved problem. Regards Johan Gill Agama Technologies From regebro at gmail.com Thu Jan 7 12:12:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 12:12:17 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45AD8C.5000405@agama.tv> References: <4B45AD8C.5000405@agama.tv> Message-ID: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I imagine that other companies have contributed before, so this is > probably an already solved problem. I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hodgestar+pythondev at gmail.com Thu Jan 7 12:28:01 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 7 Jan 2010 13:28:01 +0200 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon From swamy.sangamesh at gmail.com Thu Jan 7 11:46:59 2010 From: swamy.sangamesh at gmail.com (swamy sangamesh) Date: Thu, 7 Jan 2010 16:16:59 +0530 Subject: [Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9 Message-ID: Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have attached the full test case result. If i change the type c_int and c_short to c_unit and c_ushort of class "BITS(Structure)" in file test_bitfields.py then no failures are seen. Has anyone faced the similar issue or any help is appreciated. Thanks, Sangamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ctype-testcases Type: application/octet-stream Size: 22996 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 7 13:23:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 07 Jan 2010 22:23:11 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: <4B45D22F.40404@gmail.com> Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls we need to think >> about? I imagine that other companies have contributed before, so this is >> probably an already solved problem. > > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license without requesting additional permission from the copyright holder. For corporate contributions, I believe the contributor agreement needs to be signed by an authorised agent of the company - the place to check that would probably be psf at python.org (that's the email address for the PSF board). Assuming the subject line relates to the code that you would like to contribute though, that particular change is unlikely to happen - 2.6 is in maintenance mode and changing RLock from a Python implementation to the faster C one is solidly in new feature territory. Although a backport of the 3.2 C RLock implementation to 2.7 could be useful, I doubt that backporting code provided by an existing committer would be the subject of this query :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Thu Jan 7 14:11:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:11:27 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. Even if the contributed code as in this case is a method in an existing file? How does that work, how do they keep ownership of one method in the threading.py method? :-) > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) Ah. I probably misunderstood what the suggested contribution was. Maybe it was a separate file, which I didn't get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Thu Jan 7 14:15:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 07 Jan 2010 13:15:09 +0000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> Message-ID: <4B45DE5D.3030104@voidspace.org.uk> On 07/01/2010 13:11, Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > >> As Simon pointed out, while some organisations do work that way, the PSF >> isn't one of them. >> >> The PSF only requires that the code be contributed under a license that >> then allows us to turn around and redistribute it under a different open >> source license without requesting additional permission from the >> copyright holder. >> > Even if the contributed code as in this case is a method in an > existing file? How does that work, how do they keep ownership of one > method in the threading.py method? :-) > > When contributing code to Python all work remains copyright the original author. The combined work is copyright *everyone*. The PSF has a license to distribute it, which is all that is important. How you meaningfully exercise your ownership over chunks of code is left for the reader to determine... (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From stefan_ml at behnel.de Thu Jan 7 14:30:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:30:16 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: Guido van Rossum, 07.01.2010 05:29: > A better rule would be "you may access the memory buffer in a PyString > or PyUnicode object with the GIL released as long as you own a > reference to the string object." Everything else is out of bounds (or > not worth the bother). Is that a "yes" regarding the OP's original question about releasing the GIL during regexp searches? Stefan From regebro at gmail.com Thu Jan 7 14:36:42 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:36:42 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DE5D.3030104@voidspace.org.uk> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> <4B45DE5D.3030104@voidspace.org.uk> Message-ID: <319e029f1001070536t49f76899j111212b02c14f192@mail.gmail.com> On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Thu Jan 7 15:05:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:05:23 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB, 07.01.2010 04:07: > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. > or PyErr_SetString? Certainly not safe. > Is there an easy way to find out? Release it and fix any crashes? Note that this isn't a safe solution, though, as some GIL requiring code may be platform specific. So a better approach might be to extract any obviously problematic stuff from the existing code (such as any exception handling, explicit ref-counting or object creation), and *then* try to release the GIL. Stefan From solipsis at pitrou.net Thu Jan 7 16:38:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 15:38:31 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or macros you want to call with the GIL released, and assess whether it is safe to call them. As already pointed out, no reference count should be changed, and generally no mutable container should be accessed, except if that container is known not to be referenced anywhere else (that would be the case for e.g. a list that your function has created and is busy populating). I agree that releasing the GIL when doing non-trivial regex searches is a worthwhile research, so please don't give up immediately :-) Regards Antoine Pitrou. From olemis at gmail.com Thu Jan 7 19:24:59 2010 From: olemis at gmail.com (Olemis Lang) Date: Thu, 7 Jan 2010 13:24:59 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> Message-ID: <24ea26601001071024m2abd988fuc77f94845d57483a@mail.gmail.com> On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am strongly -1 for modifying the classes in ?traditional? unittest > module [2]_ (except that I am strongly +1 for the package structure > WITHOUT TOUCHING anything else ...) , and the more I think about it I > am more convinced ... but anyway, this not a big deal (and in the end > what I think is not that relevant either ... :o). So ... > IOW, if I had all the freedom to implement it, after the pkg structure I'd do something like : {{{ #!python class TestResult: # Everything just the same def _is_relevant_tb_level(self, tb): return '__unittest' in tb.tb_frame.f_globals class BetterTestResult(TestResult): # Further code ... maybe ;o) # def _is_relevant_tb_level(self, tb): # This or anything else you might want to do ;o) # globs = tb.tb_frame.f_globals is_relevant = '__name__' in globs and \ globs["__name__"].startswith("unittest") del globs return is_relevant }}} that's what inheritance is for ;o) ... but quite probably that's not gonna happen, just a comment . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Ubuntu sustituye GIMP por F-Spot - http://feedproxy.google.com/~r/simelo-es/~3/-g48D6T6Ojs/ubuntu-sustituye-gimp-por-f-spot.html From martin at v.loewis.de Thu Jan 7 21:24:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:24:32 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: <4B464300.2020204@v.loewis.de> >> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question about releasing the > GIL during regexp searches? No, because the regex engine may also operate on buffers that start moving around when you release the GIL. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:27:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:27:09 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B46439D.9030604@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in place - if their contents changes, regex results may be confusing. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:31:21 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Message-ID: <4B464499.4020305@v.loewis.de> > I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process a file > called ast.txt. Where does this file come from ? Is it generated from > Python.asdl ? astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. HTH, Martin From foom at fuhm.net Thu Jan 7 21:36:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 7 Jan 2010 15:36:37 -0500 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B46439D.9030604@v.loewis.de> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> On Jan 7, 2010, at 3:27 PM, Martin v. L?wis wrote: >> I've been wondering whether it's possible to release the GIL in the >> regex engine during matching. > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Even if they stay in place - > if their contents changes, regex results may be confusing. It seems probably worthwhile to optimize for the common case of using the regexp engine on an immutable object of type "str" or "bytes", and allow releasing the GIL in *that* case, even if you have to keep it for the general case. James From martin at v.loewis.de Thu Jan 7 21:45:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:45:42 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> Message-ID: <4B4647F6.2060309@v.loewis.de> >>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even if they stay in place - >> if their contents changes, regex results may be confusing. > > It seems probably worthwhile to optimize for the common case of using > the regexp engine on an immutable object of type "str" or "bytes", and > allow releasing the GIL in *that* case, even if you have to keep it for > the general case. Right. This problem was the one that I thought of first. Thinking about these things is fairly difficult (to me, at least), so I think I could only tell whether I would consider a patch thread-safe that released the GIL around matching under selected circumstances - if I had the patch available. I don't see any obvious reason (assuming Guido's list of conditions holds - i.e. you are holding references to everything you access). Regards, Martin From ncoghlan at gmail.com Thu Jan 7 21:48:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Jan 2010 06:48:05 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DECB.7070907@agama.tv> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <4B45DECB.7070907@agama.tv> Message-ID: <4B464885.40806@gmail.com> Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a contributor agreement was essential for this particular contribution would depend on how much new code was needed for the backport, but the bulk of the copyright on the C RLock code would remain with Antoine regardless. However, sorting through the legalities of the contributor agreement really is the best way to make sure every is squared away nice and neatly from a legal point of view. After all, even if I was a lawyer (which I'm not, I'm just a developer with an interest in licensing issues), I still wouldn't be *your* lawyer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Jan 7 21:43:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 20:43:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the representation isn't moved away until the buffer is released. Regards Antoine. From martin at v.loewis.de Thu Jan 7 21:59:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:59:29 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B464B31.7040406@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin From johan.gill at agama.tv Thu Jan 7 14:16:59 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 14:16:59 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <4B45DECB.7070907@agama.tv> On 01/07/2010 01:23 PM, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. For corporate contributions, I believe the contributor > agreement needs to be signed by an authorised agent of the company - the > place to check that would probably be psf at python.org (that's the email > address for the PSF board). > > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) > > Regards, > Nick. > > Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Regards Johan Gill Agama Technologies From yoann.padioleau at facebook.com Thu Jan 7 22:07:55 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Thu, 7 Jan 2010 13:07:55 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464499.4020305@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> Message-ID: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> On Jan 7, 2010, at 12:31 PM, Martin v. L?wis wrote: >> I would like to use astgen.py to generate python classes corresponding to the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take as an argument a .asdl file, but apparently it instead process a file >> called ast.txt. Where does this file come from ? Is it generated from >> Python.asdl ? > > astgen.py is not used to process asdl files; ast.txt lives right next to > astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a reflection mechanism things that were generated from ast.txt, then there could be a mismatch. Where does ast.txt comes from ? Shouldn't it be generated itself from Python.adsl ? So we would have Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py containing all the UnarySub, Expression, classes that represents a Python AST. > > HTH, > Martin From martin at v.loewis.de Thu Jan 7 22:11:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 07 Jan 2010 22:11:36 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <4B464E08.5020703@v.loewis.de> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the representation isn't moved away until the buffer is > released. In 2.7, we currently get the buffer with bf_getreadbuffer. In 3.x, we have /* Release the buffer immediately --- possibly dangerous but doing something else would require some re-factoring */ PyBuffer_Release(&view); Even if we do use the new API, and correctly, it still might be confusing if the contents of the buffer changes underneath. Regards, Martin From martin at v.loewis.de Thu Jan 7 22:16:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 22:16:12 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> Message-ID: <4B464F1C.7020404@v.loewis.de> >> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose as a reflection mechanism things that were generated from > ast.txt, then there could be a mismatch. Where does ast.txt comes > from ? Shouldn't it be generated itself from Python.adsl ? What you may not be aware of is that Tools/compiler (and the compiler package that it builds on) are both unused and unmaintained. If the package stops working correctly - tough luck. > So we would have > > Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py > containing all the UnarySub, Expression, classes that represents a > Python AST. No - what actually happens in Python 3.x is this: both the compiler package and Tools/compiler are removed. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 01:10:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 01:10:35 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM Message-ID: <201001080110.35800.victor.stinner@haypocalc.com> Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading an UTF-8 text file including a BOM: #7185 (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with the UTF-8-SIG encoding, but it's possible to do better. I propose to improve open() (TextIOWrapper) by using the BOM to choose the right encoding. I think that only files opened in read only mode should support this new feature. *Read* the BOM in a *write* only file would cause unexpected behaviours. Since my proposition changes the result TextIOWrapper.read()/readline() for files starting with a BOM, we might introduce an option to open() to enable the new behaviour. But is it really needed to keep the backward compatibility? I wrote a proof of concept attached to the issue #7651. My patch only changes the behaviour of TextIOWrapper for reading files starting with a BOM. It doesn't work yet if a seek() is used before the first read. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Jan 8 01:52:20 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 16:52:20 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner wrote: > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". > > See recent issues related to reading an UTF-8 text file including a BOM: #7185 > (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with > the UTF-8-SIG encoding, but it's possible to do better. > > I propose to improve open() (TextIOWrapper) by using the BOM to choose the > right encoding. I think that only files opened in read only mode should > support this new feature. *Read* the BOM in a *write* only file would cause > unexpected behaviours. > > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? > > I wrote a proof of concept attached to the issue #7651. My patch only changes > the behaviour of TextIOWrapper for reading files starting with a BOM. It > doesn't work yet if a seek() is used before the first read. > > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Fri Jan 8 03:23:08 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 02:23:08 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46970C.9010306@mrabarnett.plus.com> Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > Alternatively, have a universal UTF-8/16/32 encoding, ie one that expects UTF-8, with or without BOM, or UTF-16/32 with BOM. > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> >> See recent issues related to reading an UTF-8 text file including a BOM: #7185 >> (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with >> the UTF-8-SIG encoding, but it's possible to do better. >> >> I propose to improve open() (TextIOWrapper) by using the BOM to choose the >> right encoding. I think that only files opened in read only mode should >> support this new feature. *Read* the BOM in a *write* only file would cause >> unexpected behaviours. >> >> Since my proposition changes the result TextIOWrapper.read()/readline() for >> files starting with a BOM, we might introduce an option to open() to enable >> the new behaviour. But is it really needed to keep the backward compatibility? >> >> I wrote a proof of concept attached to the issue #7651. My patch only changes >> the behaviour of TextIOWrapper for reading files starting with a BOM. It >> doesn't work yet if a seek() is used before the first read. >> From nick.bastin at gmail.com Fri Jan 8 04:11:03 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 7 Jan 2010 22:11:03 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> Message-ID: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs to do this on FreeBSD as well (adding bsd to the list of platforms for which this is performed "solves" the problem, but I don't pretend to know enough about either distutils or freebsd to determine if this is the correct solution). -- Nick From glyph at twistedmatrix.com Fri Jan 8 04:34:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 7 Jan 2010 22:34:36 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? It *is* crazy, but unfortunately rather common. Wikipedia has a good description of the issues: . Basically, some Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as being UTF-8, so it's become a convention to do that. That's not good enough, so you need to guess the encoding as well to make sure, but if there is a BOM and you can otherwise verify that the file is probably UTF-8 encoded, you should discard it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Jan 8 05:17:28 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 07 Jan 2010 23:17:28 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solaris and automatically adds '.' to the library_dirs, and > I suspect it just needs to do this on FreeBSD as well (adding bsd to > the list of platforms for which this is performed "solves" the > problem, but I don't pretend to know enough about either distutils or > freebsd to determine if this is the correct solution). I wouldn't say it needed discussion on the SIG: just create a bug report, with the tentative patch you have worked out, and get it assigned to Tarek. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGsdQACgkQ+gerLs4ltQ5BMQCgtV8snMXH/6dDwgdN4sIJljLd koYAoKq6c0tKsRSrITHcygu4Od9FVzF5 =BJaE -----END PGP SIGNATURE----- From guido at python.org Fri Jan 8 05:21:04 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 20:21:04 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > > file starting with a BOM, read()/readline() returns also the BOM whereas the > > BOM should be "ignored". > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > > It *is* crazy, but unfortunately rather common. ?Wikipedia has a good > description of the issues: > . ?Basically, some > Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as > being UTF-8, so it's become a convention to do that. ?That's not good > enough, so you need to guess the encoding as well to make sure, but if there > is a BOM and you can otherwise verify that the file is probably UTF-8 > encoded, you should discard it. That doesn't make sense. If the file isn't UTF-8 you can't see the BOM, because the BOM itself is UTF-8-encoded. (And yes, I know this happens. Doesn't mean we need to auto-guess by default; there are lots of issues e.g. what should happen after seeking to offset 0?) -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Fri Jan 8 07:06:16 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 08 Jan 2010 15:06:16 +0900 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from the user for each instance. But it should do something sane when reading such files. I can't really see any harm in throwing it away, especially since use of ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated IIRC. From tseaver at palladion.com Fri Jan 8 07:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 01:12:12 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> >> BOM should be "ignored". >> >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. > > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) The BOM should not be seekeable if the file is opened with the proposed "guess encoding from BOM" mode: it isn't properly part of the stream at all in that case. A UTF-8 BOM is an absurditiy, but it exists *everywhere* in the wild: Python would do wll to make it as easy as possible to consume such files, as well as the non-insane versions (UTF-16 / UTF-32 BOMs). In the best of all possible worlds, I would just try opening the file so: f = open('/path/to/file', 'r', encoding="DWIFM") and any BOM present would set the encoding for the remainder of the stream.. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGzLsACgkQ+gerLs4ltQ5+cwCdGfycPdj6+cPfD23vH644SpHL sI0AoLGD7nfgMEJdJhBr90yjQQHfDgcJ =js+2 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Fri Jan 8 08:55:27 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 8 Jan 2010 02:55:27 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. It's just that the UTF-8 decoding of the BOM at the start of a file should be "". > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I think it's pretty clear that the BOM should still be skipped in that case ... From martin at v.loewis.de Fri Jan 8 10:05:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:05:17 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F54D.9090103@v.loewis.de> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I think what Glyph meant is this: if a file starts with the UTF-8 signature, assume it's UTF-8. Then validate the assumption against the rest of the file also, and then process it as UTF-8. If the rest clearly is not UTF-8, assume that the UTF-8 signature is bogus. I understood this proposal as a general processing guideline, not something the io library should do (but, say, a text editor). FWIW, I'm personally in favor of using the UTF-8 signature. If people consider them crazy talk, that may be because UTF-8 can't possibly have a byte order - hence I call it a signature, not the BOM. As a signature, I don't consider it crazy at all. There is a long tradition of having magic bytes in files (executable files, Postscript, PDF, ... - see /etc/magic). Having a magic byte sequence for plain text to denote the encoding is useful and helps reducing moji-bake. This is the reason it's used on Windows: notepad would normally assume that text is in the ANSI code page, and for compatibility, it can't stop doing that. So the UTF-8 signature gives them an exit strategy. Regards, Martin From martin at v.loewis.de Fri Jan 8 10:06:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:06:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <201001080110.35800.victor.stinner@haypocalc.com> <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4B46F59A.5060905@v.loewis.de> > But it should do something sane when reading such files. I can't > really see any harm in throwing it away, especially since use of > ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated > IIRC. And indeed it does, when you open the file in the utf-8-sig encoding. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 10:08:30 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 10:08:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46970C.9010306@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46970C.9010306@mrabarnett.plus.com> Message-ID: <201001081008.30904.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 03:23:08, MRAB a ?crit : > Guido van Rossum wrote: > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > > talk. And for the other two, perhaps it would make more sense to have > > a separate encoding-guessing function that takes a binary stream and > > returns a text stream wrapping it with the proper encoding? > > Alternatively, have a universal UTF-8/16/32 encoding, ie one that > expects UTF-8, > with or without BOM, or UTF-16/32 with BOM. Do you mean open(filename, encoding="BOM")? I suppose that "BOM" would be a magical value specific to read a text file (open(filename, "r")), not a real codec? Otherwise which encoding should be used for open(filename, "w", encoding="BOM")? -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Fri Jan 8 10:10:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:10:23 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F67F.60604@v.loewis.de> > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". It depends. If you use the utf-8-sig encoding, it *will* ignore the UTF-8 signature. > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? Absolutely. And there is no need to produce a new option, but instead use the existing options: define an encoding that auto-detects the encoding from the family of BOMs. Maybe you call it encoding="sniff". Regards, Martin From martin at v.loewis.de Fri Jan 8 10:11:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:11:51 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: <4B46F6D7.1050301@v.loewis.de> Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. I'm fairly skeptical that anybody on distutils SIG is interested in details of the Python build process... Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 11:27:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:27:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081127.44044.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : (...) > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I wrote a new version of my patch (version 3): * don't change the default behaviour: use open(filename, encoding="BOM") to check the BOM is there is any * fix for seek(0): always ignore the BOM * add an unit test: check that the right encoding is detect, but also the the BOM is ignored (especially after a seek(0)) BOM encoding doesn't work for writing into a file, so open(filename, "w", encoding="BOM") raises a ValueError. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:31:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:31:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081131.37492.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 01:52:20, Guido van Rossum a ?crit : > And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? I choosed to modify open()+TextIOWrapper instead of writing a new function because I would like to avoid an extra read operation (syscall) on the file. With my implementation, no specific read operation is needed to detect the BOM. The BOM is simply checked in the first _read_chunk(). -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:40:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:40:28 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Improve_open=28=29_to_support_reading?= =?iso-8859-1?q?_file_starting_with=09an_unicode_BOM?= In-Reply-To: <4B46F67F.60604@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> Message-ID: <201001081140.28124.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : > > Builtin open() function is unable to open an UTF-16/32 file starting with > > a BOM if the encoding is not specified (raise an unicode error). For an > > UTF-8 file starting with a BOM, read()/readline() returns also the BOM > > whereas the BOM should be "ignored". > > It depends. If you use the utf-8-sig encoding, it *will* ignore the > UTF-8 signature. Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to remove the BOM after the first read (much harder if you use a module like ConfigParser or csv). > > Since my proposition changes the result TextIOWrapper.read()/readline() > > for files starting with a BOM, we might introduce an option to open() to > > enable the new behaviour. But is it really needed to keep the backward > > compatibility? > > Absolutely. And there is no need to produce a new option, but instead > use the existing options: define an encoding that auto-detects the > encoding from the family of BOMs. Maybe you call it encoding="sniff". Good idea, I choosed open(filename, encoding="BOM"). -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri Jan 8 15:27:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:27:42 +0000 (UTC) Subject: [Python-Dev] GIL required for _all_ Python calls? References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: > > Even if we do use the new API, and correctly, it still might be > confusing if the contents of the buffer changes underneath. Well, no more confusing than when you compute a SHA1 hash or zlib- compress the buffer, is it? Regards Antoine From solipsis at pitrou.net Fri Jan 8 15:34:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:34:26 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any Well, I think if we implement this the default behaviour *should* be changed. It looks a bit senseless to have two different "auto-choose" options, one with encoding=None and one with encoding="BOM". Regards Antoine. From guido at python.org Fri Jan 8 16:48:44 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:48:44 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 6:27 AM, Antoine Pitrou wrote: > Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: >> >> Even if we do use the new API, and correctly, it still might be >> confusing if the contents of the buffer changes underneath. > > Well, no more confusing than when you compute a SHA1 hash or zlib- > compress the buffer, is it? That depends. Algorithms that make exactly one pass over the buffer will run fine (maybe producing a meaningless result). But the regex matcher may scan the buffer repeatedly (for backtracking purposes) and it would take a considerable analysis to prove that cannot mess up its internal data structures if the data underneath changes. (I give it a decent chance that it's fine, but since it was written without ever considering this possibility I'm not 100% sure.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:52:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:52:48 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 11:55 PM, Glyph Lefkowitz wrote: > I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. And I'm saying that it is, with as much certainty as we can ever guess the encoding of a file. > If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. ?It's just that the UTF-8 decoding of the BOM at the start of a file should be "". Sure, a Latin-1-encoded file could start with the same pattern that is a UTF-8-encoded BOM. But at that point, a UTF-16-encoded file is also valid Latin-1. The question was in the context of encoding-guessing; if we're guessing, a UTF-8-encoded BOM cannot signify anything else but UTF-8. (Ditto for UTF-16 and UTF-32 BOMs.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:54:15 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:54:15 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> >> I wrote a new version of my patch (version 3): >> >> ?* don't change the default behaviour: use open(filename, encoding="BOM") to >> check the BOM is there is any > > Well, I think if we implement this the default behaviour *should* be changed. > It looks a bit senseless to have two different "auto-choose" options, one with > encoding=None and one with encoding="BOM". Well there *are* two different auto options: use the environment variables (LANG etc.) or inspect the contents of the file. I think it would be useful to have ways to specify both. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:56:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:56:46 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 1:05 AM, "Martin v. L?wis" wrote: >>> It *is* crazy, but unfortunately rather common. ?Wikipedia has a good >>> description of the issues: >>> . ?Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. ?That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. > > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Sure. I said "crazy talk" only to stir up discussion. Which worked. :-) Also, I don't want Python's default behavior to change -- sniffing the encoding should be a separate option. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:59:45 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:59:45 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: > The BOM should not be seekeable if the file is opened with the proposed > "guess encoding from BOM" mode: ?it isn't properly part of the stream at > all in that case. This feels about right to me. There are still questions though: immediately after opening a file with a BOM, what should .tell() return? And regardless of that, .seek(0) should put the file in that same initial state. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Jan 8 17:03:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:03:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > > Well, I think if we implement this the default behaviour *should* be changed. > > It looks a bit senseless to have two different "auto-choose" options, one with > > encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Yes, perhaps. In the context of open() however I think it would be helpful to change the default. Moreover, reading the BOM is certainly much more reliable than our current guessing based on the locale or the "device encoding". Regards Antoine. From mal at egenix.com Fri Jan 8 17:25:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 17:25:22 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B475C72.1010207@egenix.com> Guido van Rossum wrote: > On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: >> Victor Stinner haypocalc.com> writes: >>> >>> I wrote a new version of my patch (version 3): >>> >>> * don't change the default behaviour: use open(filename, encoding="BOM") to >>> check the BOM is there is any >> >> Well, I think if we implement this the default behaviour *should* be changed. >> It looks a bit senseless to have two different "auto-choose" options, one with >> encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Shouldn't this encoding guessing be a separate function that you call on either a file or a seekable stream ? After all, detecting encodings is just as useful to have for non-file streams. You'd then avoid having to stuff everything into a single function call and also open up the door for more complex application specific guess work or defaults. The whole process would then have two steps: 1. guess encoding import codecs encoding = codecs.guess_file_encoding(filename) 2. open the file with the found encoding f = open(filename, encoding=encoding) For seekable streams f, you'd have: 1. guess encoding import codecs encoding = codecs.guess_stream_encoding(f) 2. wrap the stream with a reader for the found encoding reader_class = codecs.getreader(encoding) g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Jan 8 17:31:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:31:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver palladion.com> wrote: > > The BOM should not be seekeable if the file is opened with the proposed > > "guess encoding from BOM" mode: it isn't properly part of the stream at > > all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? tell() in the context of text I/O is specified to return an "opaque cookie". So whatever value it returns would probably be fine, as long as seeking to that value leaves the file in an acceptable state. Rewinding (seeking to 0) in the presence of a BOM is already reasonably supported by the TextIOWrapper object: >>> dec = codecs.getincrementaldecoder('utf-16')() >>> dec.decode(b'\xff\xfea\x00b\x00') 'ab' >>> dec.decode(b'\xff\xfea\x00b\x00') '\ufeffab' >>> >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' There are tests for this in test_io.py (test_encoded_writes, line 1929, and test_append_bom and test_seek_bom, line 2045). Regards Antoine. From python at mrabarnett.plus.com Fri Jan 8 17:47:18 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 16:47:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081127.44044.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B476196.4080409@mrabarnett.plus.com> Victor Stinner wrote: > Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : > (...) >> (And yes, I know this happens. Doesn't mean we need to auto-guess by >> default; there are lots of issues e.g. what should happen after >> seeking to offset 0?) > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any > * fix for seek(0): always ignore the BOM > * add an unit test: check that the right encoding is detect, but also the the > BOM is ignored (especially after a seek(0)) > > BOM encoding doesn't work for writing into a file, so open(filename, "w", > encoding="BOM") raises a ValueError. > I think it's similar to universal newline mode. You can tell it that you're reading UTF-something-encoded text (common forms only). The preference is UTF-8, but it could be UTF-8-sig (from Windows), or possibly UTF-16/32, which really need a BOM because there are multiple bytes per codepoint, so the bytes could be big-endian or little-endian. The BOM (or signature) tells you what the encoding is, defaulting to UTF-8 if there's none. If it subsequently raises a DecodeError, then so be it! Maybe there should also be a way of determining what encoding it decided it was, so that you can then write a new file in that same encoding. From status at bugs.python.org Fri Jan 8 18:07:13 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Jan 2010 18:07:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100108170714.014B078669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/01/10 - 01/08/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2544 open (+27) / 16937 closed (+15) / 19481 total (+42) Open issues with patches: 1017 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2509 (+27) pending 34 ( +0) Issues Created Or Reopened (43) _______________________________ Extended slicing with classic class behaves strangely 01/07/10 http://bugs.python.org/issue7532 reopened mark.dickinson patch optparse library documentation has an insignificant formatting i 01/01/10 CLOSED http://bugs.python.org/issue7618 created vazovsky patch imaplib shouldn't use cause DeprecationWarnings in 2.6 01/01/10 CLOSED http://bugs.python.org/issue7619 created djc Vim syntax highlight 01/02/10 http://bugs.python.org/issue7620 created july patch Test issue 01/02/10 CLOSED http://bugs.python.org/issue7621 created georg.brandl [patch] improve unicode methods: split() rsplit() and replace() 01/03/10 http://bugs.python.org/issue7622 created flox patch PropertyType missing in Lib/types.py 01/03/10 CLOSED http://bugs.python.org/issue7623 created wplappert isinstance(... ,collections.Callable) fails with oldstyle class 01/03/10 http://bugs.python.org/issue7624 created rgammans bytearray needs more tests for "b.some_method()[0] is not b" 01/03/10 http://bugs.python.org/issue7625 created flox patch Entity references without semicolon in HTMLParser 01/03/10 CLOSED http://bugs.python.org/issue7626 created stefan.schweizer mailbox.MH.remove() lock handling is broken 01/04/10 http://bugs.python.org/issue7627 created sraustein round() doesn't work correctly in 3.1.1 01/04/10 CLOSED http://bugs.python.org/issue7628 created bkovt Compiling with mingw32 gcc, content of variable "extra_postargs" 01/04/10 CLOSED http://bugs.python.org/issue7629 created popelkopp Strange behaviour of decimal.Decimal 01/04/10 CLOSED http://bugs.python.org/issue7630 created parmax undefined label: bltin-file-objects 01/04/10 CLOSED http://bugs.python.org/issue7631 created ezio.melotti dtoa.c: oversize b in quorem 01/04/10 http://bugs.python.org/issue7632 created skrah decimal.py: type conversion in context methods 01/04/10 http://bugs.python.org/issue7633 created skrah patch, easy next/previous links in documentation skip some sections 01/05/10 CLOSED http://bugs.python.org/issue7634 created gagenellina 19.6 xml.dom.pulldom doc: stub? 01/05/10 http://bugs.python.org/issue7635 created tjreedy Add a set update action to optparse 01/05/10 CLOSED http://bugs.python.org/issue7636 created hardkrash patch Improve 19.5. xml.dom.minidom doc 01/05/10 http://bugs.python.org/issue7637 created tjreedy Counterintuitive str.splitlines() inconsistency. 01/05/10 CLOSED http://bugs.python.org/issue7638 created vencabot_teppoo bdist_msi fails on files with long names 01/05/10 http://bugs.python.org/issue7639 created mmm77 buffered io seek() buggy 01/05/10 http://bugs.python.org/issue7640 created pakal patch Built-in Formatter accepts undocumented presentation type 01/06/10 CLOSED http://bugs.python.org/issue7641 created lrekucki [patch] Minor improvement in os.system doc 01/06/10 http://bugs.python.org/issue7642 created Val patch What is an ASCII linebreak? 01/06/10 http://bugs.python.org/issue7643 created flox bug in nntplib.body() method with possible fix 01/06/10 http://bugs.python.org/issue7644 created mdmullins easy test_distutils fails on Windows XP 01/06/10 http://bugs.python.org/issue7645 created austin987 test_telnetlib fails in Windows XP 01/06/10 http://bugs.python.org/issue7646 created austin987 Add statvfs flags to the posix module 01/06/10 http://bugs.python.org/issue7647 created ajax at redhat.com patch test_urllib2 fails on Windows if not run from C: 01/06/10 http://bugs.python.org/issue7648 created austin987 easy "u'%c' % char" broken for chars in range '\x80'-'\xFF' 01/07/10 http://bugs.python.org/issue7649 created ezio.melotti patch, easy test_uuid is invalid 01/07/10 http://bugs.python.org/issue7650 created austin987 Python3: guess text file charset using the BOM 01/07/10 http://bugs.python.org/issue7651 created haypo patch Merge C version of decimal into py3k. 01/07/10 http://bugs.python.org/issue7652 created mark.dickinson Fix description of the way the PythonPath Windows registry key w 01/07/10 CLOSED http://bugs.python.org/issue7653 created BoarGules patch, needs review [patch] Enable additional bytes and memoryview tests. 01/07/10 http://bugs.python.org/issue7654 created flox patch PEP 374 Title usage & script redirection typo fixes 01/07/10 CLOSED http://bugs.python.org/issue7655 created bab9e9 patch test_hashlib fails on some installations (specifically Neal's re 01/08/10 http://bugs.python.org/issue7656 created r.david.murray test_ctypes failure on AIX 5.3 01/08/10 http://bugs.python.org/issue7657 created mallayya patch OS X pythonw.c compile error with 10.4 or earlier deployment tar 01/08/10 http://bugs.python.org/issue7658 created ned.deily Problems with attribute assignment on object instances 01/08/10 http://bugs.python.org/issue7659 created pakal Issues Now Closed (40) ______________________ shutil fails when copying to NTFS in Linux 762 days http://bugs.python.org/issue1545 benjamin.peterson patch Test 751 days http://bugs.python.org/issue1619 georg.brandl Windows Registry issue with 2.5.2 AMD64 msi 645 days http://bugs.python.org/issue2539 loewis Extension module build fails for MinGW: missing vcvarsall.bat 618 days http://bugs.python.org/issue2698 Daniel26 optparse print_usage(),.. methods are not documented 542 days http://bugs.python.org/issue3340 ezio.melotti reference leaks in test_distutils 500 days http://bugs.python.org/issue3660 ezio.melotti patch _sha256 et al. encode to UTF-8 by default 20 days http://bugs.python.org/issue3745 lemburg 26backport Add Option to Bind to a Local IP Address in httplib.py 464 days http://bugs.python.org/issue3972 gregory.p.smith patch, easy no reference for optparse methods 388 days http://bugs.python.org/issue4635 ezio.melotti PyArg_Parse* should raise TypeError for float parsed with intege 339 days http://bugs.python.org/issue5080 mark.dickinson patch Don't use PyLong_SHIFT with _PyLong_AsScaledDouble() 282 days http://bugs.python.org/issue5576 mark.dickinson patch PyFrame_GetLineNumber 244 days http://bugs.python.org/issue5954 georg.brandl patch PyCode_NewEmpty 244 days http://bugs.python.org/issue5959 georg.brandl patch Add non-command help topics to help completion of cmd.Cmd 241 days http://bugs.python.org/issue5991 georg.brandl patch make error 231 days http://bugs.python.org/issue6067 georg.brandl no longer possible to hash arrays 229 days http://bugs.python.org/issue6071 benjamin.peterson patch zipfile: Invalid argument when opening zero-sized files 173 days http://bugs.python.org/issue6511 r.david.murray use different mechanism for pythonw on osx 127 days http://bugs.python.org/issue6834 ned.deily easy Py3k doc: "from __future__ import division" not necessary 32 days http://bugs.python.org/issue7432 ezio.melotti cPickle: stack underflow in load_pop() 31 days http://bugs.python.org/issue7455 pitrou patch crash in str.rfind() with an invalid start value 25 days http://bugs.python.org/issue7458 pitrou patch Implement fastsearch algorithm for rfind/rindex 26 days http://bugs.python.org/issue7462 ezio.melotti patch GZipFile.readline too slow 20 days http://bugs.python.org/issue7471 pitrou patch ssl module documentation: SSLSocket.unwrap description shown twi 5 days http://bugs.python.org/issue7592 georg.brandl Installation - 2 tests failed: test_cmd_line test_xmlrpc 7 days http://bugs.python.org/issue7601 ezio.melotti optparse library documentation has an insignificant formatting i 2 days http://bugs.python.org/issue7618 ezio.melotti patch imaplib shouldn't use cause DeprecationWarnings in 2.6 1 days http://bugs.python.org/issue7619 djc Test issue 0 days http://bugs.python.org/issue7621 ezio.melotti PropertyType missing in Lib/types.py 0 days http://bugs.python.org/issue7623 benjamin.peterson Entity references without semicolon in HTMLParser 2 days http://bugs.python.org/issue7626 r.david.murray round() doesn't work correctly in 3.1.1 0 days http://bugs.python.org/issue7628 benjamin.peterson Compiling with mingw32 gcc, content of variable "extra_postargs" 0 days http://bugs.python.org/issue7629 r.david.murray Strange behaviour of decimal.Decimal 0 days http://bugs.python.org/issue7630 mark.dickinson undefined label: bltin-file-objects 0 days http://bugs.python.org/issue7631 pitrou next/previous links in documentation skip some sections 0 days http://bugs.python.org/issue7634 georg.brandl Add a set update action to optparse 3 days http://bugs.python.org/issue7636 r.david.murray patch Counterintuitive str.splitlines() inconsistency. 0 days http://bugs.python.org/issue7638 flox Built-in Formatter accepts undocumented presentation type 0 days http://bugs.python.org/issue7641 eric.smith Fix description of the way the PythonPath Windows registry key w 0 days http://bugs.python.org/issue7653 r.david.murray patch, needs review PEP 374 Title usage & script redirection typo fixes 0 days http://bugs.python.org/issue7655 georg.brandl patch Top Issues Most Discussed (10) ______________________________ 22 [patch] improve unicode methods: split() rsplit() and replace() 5 days open http://bugs.python.org/issue7622 14 Test suite emits many DeprecationWarnings when -3 is enabled 91 days open http://bugs.python.org/issue7092 13 unicode_escape codec does not escape quotes 8 days open http://bugs.python.org/issue7615 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 0 days open http://bugs.python.org/issue7658 8 Merge C version of decimal into py3k. 1 days open http://bugs.python.org/issue7652 8 "u'%c' % char" broken for chars in range '\x80'-'\xFF' 2 days open http://bugs.python.org/issue7649 8 decimal.py: type conversion in context methods 4 days open http://bugs.python.org/issue7633 8 Patch for [ 735515 ] urllib2 should cache 301 redir 906 days open http://bugs.python.org/issue1755841 7 Test issue 0 days closed http://bugs.python.org/issue7621 7 Cannot use both read and readline method in same ZipExtFile obj 8 days open http://bugs.python.org/issue7610 From tseaver at palladion.com Fri Jan 8 22:09:54 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:09:54 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: >> The BOM should not be seekeable if the file is opened with the proposed >> "guess encoding from BOM" mode: it isn't properly part of the stream at >> all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? And regardless of that, .seek(0) should put the file in that > same initial state. I think the behavior should be something like: >>> f = open('/path/to/maybe-BOM-encoded-file', 'r', encoding='BOM') >>> f.tell() 0L >>> f.seek(-1) >>> f.tell() # count of unicode chars in decoded stream 45L >>> f.seek(0) >>> f.read(1) # read first unicode char decoded from stream. 'A' In other words, the BOM is not readable / seekable at all: it is invisible to the consumer of the decoded stream. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHnyIACgkQ+gerLs4ltQ6s3QCgznD+7FbUzfCbe5TS6OcoXjMg rdgAoJAMEXe2xwLCIwJaZ6XA6rVyTIAi =oXb3 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:19:10 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:19:10 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B475C72.1010207@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: > Shouldn't this encoding guessing be a separate function that you call > on either a file or a seekable stream ? > > After all, detecting encodings is just as useful to have for non-file > streams. Other stream sources typically have out-of-band ways to signal the encoding: only when reading from the filesystem do we pretty much *have* to guess, and in that case the BOM / signature is the best heuristic we have. Also, some non-file streams are not seekable, and so can't be guessed via a pre-pass. > You'd then avoid having to stuff everything into > a single function call and also open up the door for more complex > application specific guess work or defaults. > > The whole process would then have two steps: > > 1. guess encoding > > import codecs > encoding = codecs.guess_file_encoding(filename) Filename is not enough information: or do you mean that API to actually open the stream? > 2. open the file with the found encoding > > f = open(filename, encoding=encoding) > > For seekable streams f, you'd have: > > 1. guess encoding > > import codecs > encoding = codecs.guess_stream_encoding(f) > > 2. wrap the stream with a reader for the found encoding > > reader_class = codecs.getreader(encoding) > g = reader_class(f) > Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoU4ACgkQ+gerLs4ltQ5o3QCeLOJ7J91E+5f66vhgu1BUhYh4 9UgAnR2IeCd0BCsPez8ZilGNHJfhRn3Y =SoPb -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:14:59 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:14:59 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> It *is* crazy, but unfortunately rather common. Wikipedia has a good >>> description of the issues: >>> . Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. If the programmer opens the file using a "guess using the BOM" encoding, Python should *not* attempt to verify that the file is properly encoded: it should check for (and consume) any BOM, and then return a stream which uses the encoding inferred from the BOM. Any errors should be handled later, when characters are read, exactly as if the file had been opened with the same encoding guessed from the BOM. > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Agreed. Having that marker at the start of the file makes interop with other tools *much* easier. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoFMACgkQ+gerLs4ltQ73dACffwUfyh6Q9vUnKYf367QFjNcU RRMAoNuKCWEx7j+MSdTv+UjhAPynBc14 =uAX6 -----END PGP SIGNATURE----- From eric at trueblade.com Fri Jan 8 22:40:47 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 8 Jan 2010 16:40:47 -0500 (EST) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? >> >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. But what if the file were in (for example) a zip file? I think you definitely want to have access to this functionality outside of open(). Eric. From foom at fuhm.net Fri Jan 8 22:49:23 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 8 Jan 2010 16:49:23 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Jan 8, 2010, at 4:14 PM, Tres Seaver wrote: >> I understood this proposal as a general processing guideline, not >> something the io library should do (but, say, a text editor). >> >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly >> have >> a byte order - hence I call it a signature, not the BOM. As a >> signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote >> the >> encoding is useful and helps reducing moji-bake. This is the reason >> it's >> used on Windows: notepad would normally assume that text is in the >> ANSI >> code page, and for compatibility, it can't stop doing that. So the >> UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop > with > other tools *much* easier. Putting the BOM at the beginning of UTF-8 text files is not a good idea, it makes interop much *worse* on a unix system, not better. Without the BOM, most commands do the right thing with UTF-8 text. E.g. to concatenate two files: $ cat file-1 file-2 > file-3 With a BOM at the beginning of the file, it won't work right. Of course, you could modify "cat" (and every other stream processing command) to know how to consume and emit BOMs, and omit the extra one that would show up in the middle of the stream...but even that can't work; what about: $ (cat file-1; cat file-2) > file-3. Should the shell now know that when you run multiple commands, it should eat the BOM emitted from the second command? Basically, using a BOM in a utf-8 file is just not a good idea: it completely ruins interop with every standard unix tool. This is not to say that Python shouldn't have a way to read a file with a UTF-8 BOM: it just shouldn't encourage you to *write* such files. James From mal at egenix.com Fri Jan 8 22:51:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 22:51:26 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <4B47A8DE.1080401@egenix.com> Tres Seaver wrote: > M.-A. Lemburg wrote: > >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? > >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. Sure there are non-seekable file streams, but at least when using StringIO-type streams you don't have that restriction. An encoding detection function would provide more use in other cases as well, so instead of hiding away the functionality in the open() constructor, I'm suggesting to make expose it via the codecs module. Applications can then use it where necessary and also provide their own defaults, using other heuristics. >> You'd then avoid having to stuff everything into >> a single function call and also open up the door for more complex >> application specific guess work or defaults. > >> The whole process would then have two steps: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_file_encoding(filename) > > Filename is not enough information: or do you mean that API to actually > open the stream? Yes. The API would open the file, guess the encoding and then close it again. If you don't want that to happen, you could use the second API I mentioned below on the already open file. Note that this function could detect a lot more encodings with reasonably high probability than just BOM-prefixed ones, e.g. we could also add support to detect encoding declarations such as the ones we use in Python source files. >> 2. open the file with the found encoding > >> f = open(filename, encoding=encoding) > >> For seekable streams f, you'd have: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_stream_encoding(f) I forgot to mention: This API needs to position the file pointer to the start of the first data byte. >> 2. wrap the stream with a reader for the found encoding > >> reader_class = codecs.getreader(encoding) >> g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Fri Jan 8 22:59:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:59:04 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote: >>> Shouldn't this encoding guessing be a separate function that you call >>> on either a file or a seekable stream ? >>> >>> After all, detecting encodings is just as useful to have for non-file >>> streams. >> Other stream sources typically have out-of-band ways to signal the >> encoding: only when reading from the filesystem do we pretty much >> *have* to guess, and in that case the BOM / signature is the best >> heuristic we have. Also, some non-file streams are not seekable, and so >> can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). If the application expects a possibly-BOM-signature-marked file, but you pass it mismatched garbage: >>> f = open('some.zip', encoding='BOM") the error handling should be the same as if you passed any other mismatched encoding: >>> f = open('some.zip', encoding='UTF8') i.e., you discover the error when you try to read from the (non)encoded stream, not when you open it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHqpwACgkQ+gerLs4ltQ7uAACeKEc+WT4TASGcVl1Hfqe6L9La I6EAn1pJtngtLWPdothGbYB+zUabEvTW =TjBK -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Fri Jan 8 23:10:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 23:10:32 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: <201001082310.33029.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 22:40:47, Eric Smith a ?crit : > >> Shouldn't this encoding guessing be a separate function that you call > >> on either a file or a seekable stream ? > >> > >> After all, detecting encodings is just as useful to have for non-file > >> streams. > > > > Other stream sources typically have out-of-band ways to signal the > > encoding: only when reading from the filesystem do we pretty much > > *have* to guess, and in that case the BOM / signature is the best > > heuristic we have. Also, some non-file streams are not seekable, and so > > can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). FYI my patch (encoding="BOM") is implemented in TextIOWrapper, and TextIOWrapper takes a binary stream as input, not a filename. -- Victor Stinner http://www.haypocalc.com/ From yoann.padioleau at facebook.com Fri Jan 8 23:59:52 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Fri, 8 Jan 2010 14:59:52 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464F1C.7020404@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: On Jan 7, 2010, at 1:16 PM, Martin v. L?wis wrote: >>> astgen.py is not used to process asdl files; ast.txt lives right >>> next to astgen.py. Instead, the asdl file is processed by >>> Parser/asdl_c.py. >> >> Yes, I know that. That's why I asked about the relation between >> ast.txt and Python.adsl. If internally the parser uses the .adsl, but >> expose as a reflection mechanism things that were generated from >> ast.txt, then there could be a mismatch. Where does ast.txt comes >> from ? Shouldn't it be generated itself from Python.adsl ? > > What you may not be aware of is that Tools/compiler (and the > compiler package that it builds on) are both unused and unmaintained. I see. So if people want to analyze python code they have to use other tools (like rope?) ? or use reflection ? > > If the package stops working correctly - tough luck. > >> So we would have >> >> Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py >> containing all the UnarySub, Expression, classes that represents a >> Python AST. > > No - what actually happens in Python 3.x is this: both the compiler > package and Tools/compiler are removed. Ok. I will then create my own ast classes generator. Thanks. > > Regards, > Martin From g.brandl at gmx.net Sat Jan 9 00:10:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 09 Jan 2010 00:10:24 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: Am 08.01.2010 22:14, schrieb Tres Seaver: >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly have >> a byte order - hence I call it a signature, not the BOM. As a signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote the >> encoding is useful and helps reducing moji-bake. This is the reason it's >> used on Windows: notepad would normally assume that text is in the ANSI >> code page, and for compatibility, it can't stop doing that. So the UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop with > other tools *much* easier. Except if only 50% of the other tools support the signature. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jan 9 00:57:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 00:57:46 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: <4B47C67A.3020302@v.loewis.de> > I see. So if people want to analyze python code they have to use > other tools (like rope?) ? or use reflection ? Correct. One such tool might be the true Python compiler, along with the _ast module. Regards, Martin From victor.stinner at haypocalc.com Sat Jan 9 00:59:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 00:59:00 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM Message-ID: <201001090059.00848.victor.stinner@haypocalc.com> Hi, Thanks for all the answers! I will try to sum up all ideas here. (1) Change default open() behaviour or make it optional? Guido would like to add an option and keep open() unchanged. He wrote that checking for BOM and using system locale are too much different to be the same option (encoding=None). Antoine would like to check BOM by default, because both options (system locale vs checking for BOM) is the same thing. About Antoine choice (encoding=None): which file modes would check for a BOM? I would like to answer only the read only mode, but then open(filename, "r") and open(filename, "r+") would behave differently? => 1 point for Guido (encoding="BOM" is more explicit) Antoine choice has the advantage of directly support UTF-8+BOM, UTF-16 and UTF-32 for all applications and all modules using open(filename). => 1 point for Antoine (2) Check for a BOM while reading or detect it before? Everybody agree that checking BOM is an interesting option and should not be limited to open(). Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file name or a binary file-like object: it returns the encoding and seek to the file start or just after the BOM. I dislike this function because it requires extra file operations (open (optional), read() and seek()) and it doesn't work if the file is not seekable (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to avoid extra file operations. Note: I implemented the BOM check in TextIOWrapper; so it's already usable for any file-like object. (3) tell() and seek() on a text file starting with a BOM To be consistent with Antoine example: >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' TextIOWrapper: * tell() should return zero at file start, * seek(0) should go be to file start, * and the BOM should always be "ignored". I mean: with open("utf8bom.txt", encoding="BOM") as fp: assert fp.tell() == 0 text = fp.read() # no BOM here fp.seek(0) assert fp.read() == text -- About my patch: - BOM check is explicit: open(filebame, encoding="BOM") - tell() / seek(0) works as expected - BOM bytes are always skipped in read() / readlines() result Hum, I don't know if this email can be called a sum up ;-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 01:09:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 00:09:48 +0000 (UTC) Subject: [Python-Dev] Quick sum up about open() + BOM References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: Hello Victor, Victor Stinner haypocalc.com> writes: > > (1) Change default open() behaviour or make it optional? > [...] > > Antoine would like to check BOM by default, because both options (system > locale vs checking for BOM) is the same thing. To be clear, I am not saying it is the same thing. What I think is that it would be a mistake to use a mildly unreliable heuristic by default (the locale + device encoding heuristic) but refuse to trust a more reliable heuristic (the BOM-based detection algorithm). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Jan 9 01:14:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 00:14:39 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47CA6F.5000607@voidspace.org.uk> On 09/01/2010 00:09, Antoine Pitrou wrote: > Hello Victor, > > Victor Stinner haypocalc.com> writes: > >> (1) Change default open() behaviour or make it optional? >> >> > [...] > >> Antoine would like to check BOM by default, because both options (system >> locale vs checking for BOM) is the same thing. >> > To be clear, I am not saying it is the same thing. What I think is that it would > be a mistake to use a mildly unreliable heuristic by default (the locale + > device encoding heuristic) but refuse to trust a more reliable heuristic (the > BOM-based detection algorithm). > I concur. On Windows both UTF-8 and signature are very common, yet the platform default is the truly awful CP1252. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From floris.bruynooghe at gmail.com Sat Jan 9 01:26:05 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 9 Jan 2010 00:26:05 +0000 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B46F6D7.1050301@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> <4B46F6D7.1050301@v.loewis.de> Message-ID: <20100109002605.GB13536@laurie.devork> On Fri, Jan 08, 2010 at 10:11:51AM +0100, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > I think this problem probably needs to move over to distutils-sig, as > > it doesn't seem to be specific to the way that Python itself uses > > distutils. > > I'm fairly skeptical that anybody on distutils SIG is interested in > details of the Python build process... Uh, hum. Unfounded skepticism. ;-) But as said filing a bug sounds better in this case. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From v+python at g.nevcal.com Sat Jan 9 01:47:38 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 16:47:38 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47D22A.8070305@g.nevcal.com> On approximately 1/8/2010 3:59 PM, came the following characters from the keyboard of Victor Stinner: > Hi, > > Thanks for all the answers! I will try to sum up all ideas here. One concern I have with this implementation encoding="BOM" is that if there is no BOM it assumes UTF-8. That is probably a good assumption in some circumstances, but not in others. * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE encoded files include a BOM. It is only required that UTF-16 and UTF-32 (cases where the endianness is unspecified) contain a BOM. Hence, it might be that someone would expect a UTF-16LE (or any of the formats that don't require a BOM, rather than UTF-8), but be willing to accept any BOM-discriminated format. * Potentially, this could be expanded beyond the various Unicode encodings... one could envision that a program whose data files historically were in any particular national language locale, could want to be enhance to accept Unicode, and could declare that they will accept any BOM-discriminated format, but want to default, in the absence of a BOM, to the original national language locale that they historically accepted. That would provide a migration path for their old data files. So the point is, that it might be nice to have "BOM-otherEncodingForDefault" for each other encoding that Python supports. Not sure that is the right API, but I think it is expressive enough to handle the cases above. Whether the cases solve actual problems or not, I couldn't say, but they seem like reasonable cases. It would, of course, be nicest if OS metadata had been invented way back when, for all OSes, such that all text files were flagged with their encoding... then languages could just read the encoding and do the right thing! But we live in the real world, instead. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From python at mrabarnett.plus.com Sat Jan 9 02:12:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 09 Jan 2010 01:12:28 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B47D7FC.4070703@mrabarnett.plus.com> Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. > > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. > What about listing the possible encodings? It would try each in turn until it found one where the BOM matched or had no BOM: my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') or is that taking it too far? From martin at v.loewis.de Sat Jan 9 02:23:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 02:23:07 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47CA6F.5000607@voidspace.org.uk> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> Message-ID: <4B47DA7B.6050505@v.loewis.de> >>> Antoine would like to check BOM by default, because both options >>> (system locale vs checking for BOM) is the same thing. >>> >> To be clear, I am not saying it is the same thing. What I think is >> that it would be a mistake to use a mildly unreliable heuristic by >> default (the locale + device encoding heuristic) but refuse to >> trust a more reliable heuristic (the BOM-based detection >> algorithm). >> > > I concur. On Windows both UTF-8 and signature are very common, yet > the platform default is the truly awful CP1252. While I would support combining BOM detection in the case where a file is opened for reading and no encoding is specified, I see two problems: a) if a seek operations is performed before having looked at the BOM, no determination would have been made b) what encoding should it use on writing? Regards, Martin From v+python at g.nevcal.com Sat Jan 9 02:49:12 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 17:49:12 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <4B47E098.6030703@g.nevcal.com> On approximately 1/8/2010 5:12 PM, came the following characters from the keyboard of MRAB: > Glenn Linderman wrote: >> On approximately 1/8/2010 3:59 PM, came the following characters from >> the keyboard of Victor Stinner: >>> Hi, >>> >>> Thanks for all the answers! I will try to sum up all ideas here. >> >> One concern I have with this implementation encoding="BOM" is that if >> there is no BOM it assumes UTF-8. That is probably a good assumption >> in some circumstances, but not in others. >> >> * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE >> encoded files include a BOM. It is only required that UTF-16 and >> UTF-32 (cases where the endianness is unspecified) contain a BOM. >> Hence, it might be that someone would expect a UTF-16LE (or any of >> the formats that don't require a BOM, rather than UTF-8), but be >> willing to accept any BOM-discriminated format. >> >> * Potentially, this could be expanded beyond the various Unicode >> encodings... one could envision that a program whose data files >> historically were in any particular national language locale, could >> want to be enhance to accept Unicode, and could declare that they >> will accept any BOM-discriminated format, but want to default, in the >> absence of a BOM, to the original national language locale that they >> historically accepted. That would provide a migration path for their >> old data files. >> >> So the point is, that it might be nice to have >> "BOM-otherEncodingForDefault" for each other encoding that Python >> supports. Not sure that is the right API, but I think it is >> expressive enough to handle the cases above. Whether the cases solve >> actual problems or not, I couldn't say, but they seem like reasonable >> cases. >> >> It would, of course, be nicest if OS metadata had been invented way >> back when, for all OSes, such that all text files were flagged with >> their encoding... then languages could just read the encoding and do >> the right thing! But we live in the real world, instead. >> > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? That sounds very flexible -- but in net effect it would only make illegal a subset of the BOM-containing encodings (those not listed) without making legal any additional encodings other than the non-BOM encoding. Whether prohibiting a subset of BOM-containing encodings is a useful use case, I couldn't say... but my goal would be to included as many different file encodings on input as possible: without a BOM, that is exactly 1 (unless there are other heuristics), with a BOM, it is 1+all-BOM-containing encodings. Your scheme would permit numbers of encodings accepted to vary between 1 and 1+all-BOM-containing encodings. (I think everyone can agree there are 5 different byte sequences that can be called a Unicode BOM. The likelihood of them appearing in any other text encoding created by mankind depends on those other encodings -- but it is not impossible. It is truly up to the application to decide whether BOM detection could potentially conflict with files in some other encoding that would be acceptable to the application.) So I think it is taking it further than I can see value in, but I'm willing to be convinced otherwise. I see only a need for detecting BOM, and specifying a default encoding to be used if there is no BOM. Note that it might be nice to have a specification for using current encoding=None heuristic -- perhaps encoding="BOM-None" per my originally proposed syntax. But I'm still not saying that is the best syntax. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ncoghlan at gmail.com Sat Jan 9 04:07:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 09 Jan 2010 13:07:12 +1000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B476196.4080409@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B476196.4080409@mrabarnett.plus.com> Message-ID: <4B47F2E0.7090403@gmail.com> MRAB wrote: > Maybe there should also be a way of determining what encoding it decided > it was, so that you can then write a new file in that same encoding. I thought of that question as well - the f.encoding attribute on the opened file should be sufficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Sat Jan 9 06:48:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 06:48:36 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47E098.6030703@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> <4B47E098.6030703@g.nevcal.com> Message-ID: <319e029f1001082148h5cee2c0bn76f70a17766ca69e@mail.gmail.com> It seems to me that when opening a file, the following is the only flow that makes sense for the typical opening of a file flow: if encoding is not None: use encoding elif file has BOM: use BOM else: use system default And hence a encoding='BOM' isn't needed there. Although I'm trying to come up with usecases that doesn't work with this, I can't. :) BUT When writing things are not so easy though. Apparently some encodings require a BOM to be written, but others do not, but allow it, and some has no byte order mark. So there you have to be able to write the BOM, or not. And that's either a new parameter, because you can't use encoding='BOM' since you need to specify the encoding as well, or a new method. I would suggest a BOM parameter, and maybe a method as well. BOM=None|True|False Where "None" means a sane default behaviour, that is write a BOM if the encoding require it. "True" means write a BOM if the encoding *supports* it. "False" means Don't write a BOM even if the encoding requires it (because I know what I'm doing) if 'w' in mode: # But not 'r' or 'a' if BOM == True and encoding in (ENCODINGS THAT ALLOW BOM): write_bom = True elif BOM == False: write_bom = False elif BOM == None and encoding in (ENCODINGS THAT REQUIRE BOM): write_bom = True else: write_bom = False else: write_bom = False For reading this parameter could either be a noop, or possibly change the behavior somehow, if a usecase where that makes sense can be imagined. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From walter at livinglogic.de Sat Jan 9 11:51:57 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 11:51:57 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B485FCD.2040901@livinglogic.de> On 09.01.10 01:47, Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. This is doable with the currect API. Simply define a codec search function that handles all encoding names that start with "BOM-" and pass the "otherEncodingForDefault" part along to the codec. > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. Servus, Walter From walter at livinglogic.de Sat Jan 9 12:18:33 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 12:18:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081140.28124.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> Message-ID: <4B486609.2050804@livinglogic.de> Victor Stinner wrote: > Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : >>> Builtin open() function is unable to open an UTF-16/32 file starting with >>> a BOM if the encoding is not specified (raise an unicode error). For an >>> UTF-8 file starting with a BOM, read()/readline() returns also the BOM >>> whereas the BOM should be "ignored". >> It depends. If you use the utf-8-sig encoding, it *will* ignore the >> UTF-8 signature. > > Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and > UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to > remove the BOM after the first read (much harder if you use a module like > ConfigParser or csv). > >>> Since my proposition changes the result TextIOWrapper.read()/readline() >>> for files starting with a BOM, we might introduce an option to open() to >>> enable the new behaviour. But is it really needed to keep the backward >>> compatibility? >> Absolutely. And there is no need to produce a new option, but instead >> use the existing options: define an encoding that auto-detects the >> encoding from the family of BOMs. Maybe you call it encoding="sniff". > > Good idea, I choosed open(filename, encoding="BOM"). On the surface this looks like there's an encoding named "BOM", but looking at your patch I found that the check is still done in TextIOWrapper. IMHO the best approach would to the implement a *real* codec named "BOM" (or "sniff"). This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop. To see how something like this can be done, take a look at the UTF-16 codec, that switches to bigendian or littleendian mode depending on the first read/decode call. Servus, Walter From victor.stinner at haypocalc.com Sat Jan 9 13:37:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:37:06 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47DA7B.6050505@v.loewis.de> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> <4B47DA7B.6050505@v.loewis.de> Message-ID: <201001091337.06596.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:23:07, Martin v. L?wis a ?crit : > While I would support combining BOM detection in the case where a file > is opened for reading and no encoding is specified, I see two problems: > a) if a seek operations is performed before having looked at the BOM, > no determination would have been made TextIOWrapper doesn't support seek to an arbitrary byte. It uses "cookie" which is an opaque value. Reuse a cookie from another file or an old cookie is forbidden (but it doesn't raise an error). This is not specific to the BOM checking: the problem already exist for encodings using a BOM (eg. UTF-16). > b) what encoding should it use on writing? Don't change anything to writing. With Antoince choice: open('file.txt', 'w', encoding=None) continue to use the actual heuristic (os.device_encoding() or system locale). With Guido choice, encoding="BOM": it raises an error, because BOM check is not supported when writing into a file. How could the BOM be checked when creating a new (empty) file!? -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Sat Jan 9 13:45:58 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 09 Jan 2010 13:45:58 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B487A86.4020603@egenix.com> Victor Stinner wrote: > (2) Check for a BOM while reading or detect it before? > > Everybody agree that checking BOM is an interesting option and should not be > limited to open(). > > Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file > name or a binary file-like object: it returns the encoding and seek to the > file start or just after the BOM. > > I dislike this function because it requires extra file operations (open > (optional), read() and seek()) and it doesn't work if the file is not seekable > (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to > avoid extra file operations. > > Note: I implemented the BOM check in TextIOWrapper; so it's already usable for > any file-like object. Yes, but the implementation is limited to just BOM checking and thus only supports UTF-8-SIG, UTF-16 and UTF-32. With a codecs module function we could easily extend the encoding detection to more file types, e.g. XML files, Python source code files, etc. that use other mechanisms for defining the encoding. BTW: I haven't looked at your implementation, but what happens when your BOM check fails ? Will the implementation add the already read bytes back to a buffer ? This rollback action is the only reason for needing a seekable stream in codecs.guess_stream_encoding(). Another point to consider: AFAIK, we currently have a moratorium on changes to Python builtins. How does that match up with the proposed changes ? Using a new codec like Walter suggested would move the implementation into the stdlib for which doesn't the moratorium doesn't apply. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Sat Jan 9 13:54:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:54:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <201001091354.17239.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 01:47:38, vous avez ?crit : > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. If no BOM is found, it fallback to the current heuristic: os.device_encoding() or system local. > (...) Hence, it might be that someone would expect a UTF-16LE (or any of > the formats that don't require a BOM, rather than UTF-8), but be willing > to accept any BOM-discriminated format. > (...) declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted You mean "if there is a BOM, use it, otherwise fallback to a specific charset"? How could it be declared? Maybe: open("file.txt", check_bom=True, encoding="UTF16-LE") open("file.txt", check_bom=True, encoding="latin1") About falling back to UTF-8, it would be written: open("file.txt", check_bom=True, encoding="UTF-8") As explained before, check_bom=True is only accepted for read only file mode. Well, why not. This is a third choice for my point (1) :-) It's between Guido and Antoine choice, and I like it because we can fallback to UTF-8 instead of the dummy system locale: Windows users will be happy to be able to use UTF-8 :-) I prefer to fallback to a fixed encoding then depending on the system locale. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:34:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:34:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <201001091434.17521.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:12:28, MRAB a ?crit : > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? Yes, you're taking it foo far :-) Checking BOM is reliable, whereas *guessing* the charset only using the byte stream can only be an heuristic. Guess a charset is a complex problem, they are 3rd party library to do that, like the chardet project. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:38:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:38:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B486609.2050804@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <201001091438.43576.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : > > Good idea, I choosed open(filename, encoding="BOM"). > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Why not, this is another solution to the point (2) (Check for a BOM while reading or detect it before?). Which encoding would be used if there is not BOM? UTF-8 sounds like a good choice. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:50:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:50:28 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B487A86.4020603@egenix.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B487A86.4020603@egenix.com> Message-ID: <201001091450.28497.victor.stinner@haypocalc.com> Hi, Le samedi 09 janvier 2010 13:45:58, vous avez ?crit : > > Note: I implemented the BOM check in TextIOWrapper; so it's already > > usable for any file-like object. > > Yes, but the implementation is limited to just BOM checking > and thus only supports UTF-8-SIG, UTF-16 and UTF-32. Sure, but that's already better than no BOM check :-) It looks like many people would apprecite UTF-8-SIG detection, since this encoding is common on Windows. > BTW: I haven't looked at your implementation, but what happens > when your BOM check fails ? Will the implementation add the > already read bytes back to a buffer ? My implementation is done between buffer.read() and decoder.decode(data). If there is a BOM: set the encoding and remove the BOM bytes from the byte string. Otherwise, use another algorithm to choose the encoding and leave the byte string unchanged. It can be seen as a codec: it works like UTF-16 and UTF-32 codecs ;-) > AFAIK, we currently have a moratorium on changes to Python > builtins. How does that match up with the proposed changes ? Oh yes, I forgot the moratorium. In all solutions, some of them don't change the API. Eg. Antoine proposed to leave the API unchanged: open(file) => open(file) :-) I don't know if it's compatible with the moratorium or not. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 16:05:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 15:05:45 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: Walter D?rwald livinglogic.de> writes: > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Sorry but this is missing the point. The point here is to improve the open() function. I'm sure people who know about encodings are able to install the chardet library or even whip up their own BOM detection routine... Antoine. From benjamin at python.org Sat Jan 9 18:29:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 11:29:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 Message-ID: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> On behalf of the Python development team, I'm gleeful to announce the second alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From kmtracey at gmail.com Sat Jan 9 19:48:12 2010 From: kmtracey at gmail.com (Karen Tracey) Date: Sat, 9 Jan 2010 13:48:12 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce the > second > alpha release of Python 2.7. > > Well yay. Django's test suite (1242 tests) runs with just one failure on the 2.7 alpha 2 level, and that looks to be likely due to the improved string/float rounding so not really a problem, just a difference. That's down from 104 failures and 40 errors with 2.7 alpha 1. Note on the website page http://www.python.org/download/releases/2.7/ the "Change log for this release" link is still pointing to the alpha 1 changelog. Thanks, Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jan 9 19:51:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 12:51:11 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <1afaf6161001091051kb1ba08q9b96094af16c12fa@mail.gmail.com> 2010/1/9 Karen Tracey : > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson > wrote: >> >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> > > Well yay.? Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference.? That's > down from 104 failures and 40 errors with 2.7 alpha 1. Excellent! > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Thanks. I'll fix that. > -- Regards, Benjamin From skip at pobox.com Sat Jan 9 21:00:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:00:44 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 Message-ID: <19272.57452.972234.643008@montanaro.dyndns.org> How much of the Unladen Swallow cPickle speedups have been incorporated into 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the two versions I currently care about at work - we will skip 2.5 entirely). It appears some of their speedups may have already been merged to trunk, but I'm not sure how much. If a patch to merge this to 2.7 is already under consideration I won't look at it, but I interpreted Collin Winter's response to my query on the u-s mailing list that not everything has been done yet. Thx, Skip From martin at v.loewis.de Sat Jan 9 21:09:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 09 Jan 2010 21:09:27 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <4B48E277.9010408@v.loewis.de> Antoine Pitrou wrote: > Walter D?rwald livinglogic.de> writes: >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Sorry but this is missing the point. The point here is to improve the open() > function. I'm sure people who know about encodings are able to install the > chardet library or even whip up their own BOM detection routine... How does the requirement that it be implemented as a codec miss the point? FWIW, I agree with Walter that if it is provided through the encoding= argument, it should be a codec. If it is built into the open function (for whatever reason), it must be provided by some other parameter. I do see the point that it becomes available to end users only when released as part of Python. However, this *also* means that applications won't be using it for another three years or so, since they'll have to support older Python versions as well (unless it is integrated in the case where no encoding is specified). Regards, Martin From pjenvey at underboss.org Sat Jan 9 21:09:29 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 9 Jan 2010 12:09:29 -0800 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <19272.57452.972234.643008@montanaro.dyndns.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> On Jan 9, 2010, at 12:00 PM, skip at pobox.com wrote: > How much of the Unladen Swallow cPickle speedups have been incorporated into > 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the > two versions I currently care about at work - we will skip 2.5 entirely). > It appears some of their speedups may have already been merged to trunk, but > I'm not sure how much. If a patch to merge this to 2.7 is already under > consideration I won't look at it, but I interpreted Collin Winter's response > to my query on the u-s mailing list that not everything has been done yet. They've documented their upstream patches here: http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches -- Philip Jenvey From skip at pobox.com Sat Jan 9 21:21:20 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:21:20 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> Message-ID: <19272.58688.173011.412712@montanaro.dyndns.org> Philip> They've documented their upstream patches here: Philip> http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches Thanks. That will help immensely. Skip From solipsis at pitrou.net Sat Jan 9 21:28:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:28:12 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Sorry but this is missing the point. The point here is to improve the open() > > function. I'm sure people who know about encodings are able to install the > > chardet library or even whip up their own BOM detection routine... > > How does the requirement that it be implemented as a codec miss the > point? If we want it to be the default, it must be able to fallback on the current locale-based algorithm if no BOM is found. I don't think it would be easy for a codec to do that. > FWIW, I agree with Walter that if it is provided through the encoding= > argument, it should be a codec. If it is built into the open function > (for whatever reason), it must be provided by some other parameter. Why not simply encoding=None? The default value should provide the most useful behaviour possible. Forcing users to choose between two different autodetection strategies (encoding=None and another one) is a little insane IMO. Regards Antoine. From solipsis at pitrou.net Sat Jan 9 21:32:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:32:27 +0000 (UTC) Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > If a patch to merge this to 2.7 is already under > consideration I won't look at it, Why won't you look at it? :) Actually, if these patches are to be merged someone should certainly look at them, and do the (possibly) remaining work. http://bugs.python.org/issue5683 http://bugs.python.org/issue5671 Thank you Antoine. From skip at pobox.com Sat Jan 9 21:43:42 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:43:42 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <19272.60030.25319.269597@montanaro.dyndns.org> >>>>> "Antoine" == Antoine Pitrou writes: Antoine> pobox.com> writes: >> >> If a patch to merge this to 2.7 is already under >> consideration I won't look at it, Antoine> Why won't you look at it? :) I meant I wouldn't look at developing one. Skip From regebro at gmail.com Sat Jan 9 23:14:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 23:14:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Right. It seems like encoding=None is the right way to go there. encoding='BOM' would probably only work if 'BOM' isn't an encoding but a special tag, which is ugly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Sun Jan 10 00:25:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 23:25:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> Message-ID: <4B49105E.303@voidspace.org.uk> On 09/01/2010 22:14, Lennart Regebro wrote: > On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. >> > Right. It seems like encoding=None is the right way to go there. > encoding='BOM' would probably only work if 'BOM' isn't an encoding but > a special tag, which is ugly. > > I would rather see it as the default behavior for open without an encoding specified. I know Guido has expressed a preference against this so I won't continue to flog it. The current behavior however is that we have a 'guessing' algorithm based on the platform default. Currently if you open a text file in read mode that has a UTF-8 signature, but the platform default is something other than UTF-8, then we open the file using what is likely to be the incorrect encoding. Looking for the signature seems to be better behaviour in that case. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jan 10 00:40:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 10 Jan 2010 00:40:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <4B4913DF.7050801@v.loewis.de> >> How does the requirement that it be implemented as a codec miss the >> point? > > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Yes - however, Victor currently apparently *doesn't* want it to be the default, but wants the user to specify encoding="BOM". If so, it isn't the default, and it is easy to implement as a codec. >> FWIW, I agree with Walter that if it is provided through the encoding= >> argument, it should be a codec. If it is built into the open function >> (for whatever reason), it must be provided by some other parameter. > > Why not simply encoding=None? I don't mind. Please re-read Walter's message - it only said that *if* this is activated through encoding="BOM", *then* it must be a codec, and could be on PyPI. I don't think Walter was talking about the case "it is not activated through encoding='BOM'" *at all*. > The default value should provide the most useful > behaviour possible. Forcing users to choose between two different autodetection > strategies (encoding=None and another one) is a little insane IMO. That wouldn't disturb me much. There are a lot of things in that area that are a little insane, starting with Microsoft Windows :-) Regards, Martin From henning.vonbargen at arcor.de Sun Jan 10 12:10:02 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Sun, 10 Jan 2010 12:10:02 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: Message-ID: <4B49B58A.4040103@arcor.de> If Python should support BOM when reading text files, it should also be able to *write* such files. An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what? That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values: None: default (old) behaviour: don't handle BOM at all True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument. (*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False. Henning P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway... From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brett at python.org Sun Jan 10 19:51:26 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 10:51:26 -0800 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: Nick Coghlan thought I should forward this to python-dev so people are aware of this change and why it occurred (plus it indirectly informs Guido I finally finished the work). ---------- Forwarded message ---------- From: brett.cannon Date: Sat, Jan 9, 2010 at 18:56 Subject: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c To: python-checkins at python.org Author: brett.cannon Date: Sun Jan 10 03:56:19 2010 New Revision: 77402 Log: DeprecationWarning is now silent by default. This was originally suggested by Guido, discussed on the stdlib-sig mailing list, and given the OK by Guido directly to me. What this change essentially means is that Python has taken a policy of silencing warnings that are only of interest to developers by default. This should prevent users from seeing warnings which are triggered by an application being run against a new interpreter before the app developer has a chance to update their code. Closes issue #7319. Thanks to Antoine Pitrou, Ezio Melotti, and Brian Curtin for helping with the issue. Modified: python/trunk/Doc/library/warnings.rst python/trunk/Lib/test/test_ascii_formatd.py python/trunk/Lib/test/test_exceptions.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS python/trunk/Python/_warnings.c Modified: python/trunk/Doc/library/warnings.rst ============================================================================== --- python/trunk/Doc/library/warnings.rst (original) +++ python/trunk/Doc/library/warnings.rst Sun Jan 10 03:56:19 2010 @@ -59,7 +59,7 @@ | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,14 +151,6 @@ :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - .. _warning-suppress: @@ -226,6 +221,37 @@ entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions Modified: python/trunk/Lib/test/test_ascii_formatd.py ============================================================================== --- python/trunk/Lib/test/test_ascii_formatd.py (original) +++ python/trunk/Lib/test/test_ascii_formatd.py Sun Jan 10 03:56:19 2010 @@ -4,6 +4,7 @@ import unittest from test_support import check_warnings, run_unittest, cpython_only +import warnings class FormatDeprecationTests(unittest.TestCase): @@ -17,6 +18,7 @@ buf = create_string_buffer(' ' * 100) with check_warnings() as w: + warnings.simplefilter('default') PyOS_ascii_formatd(byref(buf), sizeof(buf), '%+.10f', c_double(10.0)) self.assertEqual(buf.value, '+10.0000000000') Modified: python/trunk/Lib/test/test_exceptions.py ============================================================================== --- python/trunk/Lib/test/test_exceptions.py (original) +++ python/trunk/Lib/test/test_exceptions.py Sun Jan 10 03:56:19 2010 @@ -309,6 +309,7 @@ # BaseException.__init__ triggers a deprecation warning. exc = BaseException("foo") with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('default') self.assertEquals(exc.message, "foo") self.assertEquals(len(w), 1) self.assertEquals(w[0].category, DeprecationWarning) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Sun Jan 10 03:56:19 2010 @@ -383,8 +383,8 @@ # Module initialization _processoptions(sys.warnoptions) if not _warnings_defaults: - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) + for cls in (DeprecationWarning, PendingDeprecationWarning, ImportWarning): + simplefilter("ignore", category=cls, append=True) bytes_warning = sys.flags.bytes_warning if bytes_warning > 1: bytes_action = "error" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Jan 10 03:56:19 2010 @@ -12,6 +12,8 @@ Core and Builtins ----------------- +- Issue #7319: Silence DeprecationWarning by default. + - Issue #2335: Backport set literals syntax from Python 3.x. Library Modified: python/trunk/Python/_warnings.c ============================================================================== --- python/trunk/Python/_warnings.c (original) +++ python/trunk/Python/_warnings.c Sun Jan 10 03:56:19 2010 @@ -85,10 +85,10 @@ default_action = get_warnings_attr("defaultaction"); if (default_action == NULL) { - if (PyErr_Occurred()) { - return NULL; - } - return _default_action; + if (PyErr_Occurred()) { + return NULL; + } + return _default_action; } Py_DECREF(_default_action); @@ -202,12 +202,12 @@ mod_str = PyString_AsString(filename); if (mod_str == NULL) - return NULL; + return NULL; len = PyString_Size(filename); if (len < 0) return NULL; if (len >= 3 && - strncmp(mod_str + (len - 3), ".py", 3) == 0) { + strncmp(mod_str + (len - 3), ".py", 3) == 0) { module = PyString_FromStringAndSize(mod_str, len-3); } else { @@ -251,7 +251,7 @@ name = PyObject_GetAttrString(category, "__name__"); if (name == NULL) /* XXX Can an object lack a '__name__' attribute? */ - return; + return; f_stderr = PySys_GetObject("stderr"); if (f_stderr == NULL) { @@ -341,7 +341,7 @@ rc = already_warned(registry, key, 0); if (rc == -1) goto cleanup; - else if (rc == 1) + else if (rc == 1) goto return_none; /* Else this warning hasn't been generated before. */ } @@ -492,9 +492,9 @@ /* Setup filename. */ *filename = PyDict_GetItemString(globals, "__file__"); if (*filename != NULL) { - Py_ssize_t len = PyString_Size(*filename); + Py_ssize_t len = PyString_Size(*filename); const char *file_str = PyString_AsString(*filename); - if (file_str == NULL || (len < 0 && PyErr_Occurred())) + if (file_str == NULL || (len < 0 && PyErr_Occurred())) goto handle_error; /* if filename.lower().endswith((".pyc", ".pyo")): */ @@ -506,10 +506,10 @@ tolower(file_str[len-1]) == 'o')) { *filename = PyString_FromStringAndSize(file_str, len-1); - if (*filename == NULL) - goto handle_error; - } - else + if (*filename == NULL) + goto handle_error; + } + else Py_INCREF(*filename); } else { @@ -536,8 +536,8 @@ else { /* embedded interpreters don't have sys.argv, see bug #839151 */ *filename = PyString_FromString("__main__"); - if (*filename == NULL) - goto handle_error; + if (*filename == NULL) + goto handle_error; } } if (*filename == NULL) { @@ -839,26 +839,29 @@ static PyObject * init_filters(void) { - PyObject *filters = PyList_New(3); + PyObject *filters = PyList_New(4); const char *bytes_action; if (filters == NULL) return NULL; PyList_SET_ITEM(filters, 0, + create_filter(PyExc_DeprecationWarning, "ignore")); + PyList_SET_ITEM(filters, 1, create_filter(PyExc_PendingDeprecationWarning, "ignore")); - PyList_SET_ITEM(filters, 1, create_filter(PyExc_ImportWarning, "ignore")); + PyList_SET_ITEM(filters, 2, create_filter(PyExc_ImportWarning, "ignore")); if (Py_BytesWarningFlag > 1) bytes_action = "error"; else if (Py_BytesWarningFlag) bytes_action = "default"; else bytes_action = "ignore"; - PyList_SET_ITEM(filters, 2, create_filter(PyExc_BytesWarning, + PyList_SET_ITEM(filters, 3, create_filter(PyExc_BytesWarning, bytes_action)); if (PyList_GET_ITEM(filters, 0) == NULL || PyList_GET_ITEM(filters, 1) == NULL || - PyList_GET_ITEM(filters, 2) == NULL) { + PyList_GET_ITEM(filters, 2) == NULL || + PyList_GET_ITEM(filters, 3) == NULL) { Py_DECREF(filters); return NULL; } _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sun Jan 10 20:22:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 10 Jan 2010 20:22:10 +0100 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: <201001102022.10259.victor.stinner@haypocalc.com> Hi, Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : > Nick Coghlan thought I should forward this to python-dev so people are > aware of this change and why it occurred (plus it indirectly informs Guido > I finally finished the work). Thanks to copy this mail to the python-dev mailing list. What is the recommanded way to get the previous behaviour (print deprecation warnings once)? Is there a command line option? Is it "-W default"? -- Victor Stinner http://www.haypocalc.com/ From benjamin at python.org Sun Jan 10 20:23:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 13:23:54 -0600 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <201001102022.10259.victor.stinner@haypocalc.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> <201001102022.10259.victor.stinner@haypocalc.com> Message-ID: <1afaf6161001101123g366d80dbjeaa80f1a632605e2@mail.gmail.com> 2010/1/10 Victor Stinner : > Hi, > > Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : >> Nick Coghlan thought I should forward this to python-dev so people are >> ?aware of this change and why it occurred (plus it indirectly informs Guido >> ?I finally finished the work). > > Thanks to copy this mail to the python-dev mailing list. > > What is the recommanded way to get the previous behaviour (print deprecation > warnings once)? Is there a command line option? Is it "-W default"? That commit conveniently adds documentation answering that question. :) -- Regards, Benjamin From nas at arctrix.com Sun Jan 10 20:30:09 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:30:09 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce > the second alpha release of Python 2.7. Thanks to everyone who contributed. > Python 2.7 is scheduled to be the last major version in the 2.x > series. Has this really been decided already? Maybe I missed it. In my opinion, it does Python's reputation harm to make such a statement. Conservative users (probably the vast majority of Python users) don't like to hear that software they are considering using is nearing the end of its life. What does it gain us to announce that the 2.x branch is dead aside from bugfixes? I propose that the 2.x branch be treated like 2.x.y branches: as long as there is sufficient volunteer labour, it should continue to live. In order to avoid wasted development effort, it would be prudent to announce that unless a 2.8 release manager steps up, whatever is committed to the 2.x branch after the 2.7 release may never get released. Said another way, it's okay for the Python developers to decide to abandon 2.x and put their efforts into 3.x. It's not okay for them to prevent others from continuing to work on 2.x or to somehow make 2.x look worse so 3.x looks better. Python 3 needs to stand on its own terms and I'm confident it can. Best regards, Neil From brett at python.org Sun Jan 10 21:09:08 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:09:08 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sun, Jan 10, 2010 at 11:30, Neil Schemenauer wrote: > Benjamin Peterson wrote: > > On behalf of the Python development team, I'm gleeful to announce > > the second alpha release of Python 2.7. > > Thanks to everyone who contributed. > > > Python 2.7 is scheduled to be the last major version in the 2.x > > series. > > Has this really been decided already? Maybe I missed it. More or less. It was first discussed at the language summit last year and has come up here a couple of times. If needed we can make it official in terms of lifetime of 2.7, etc. at the language summit this year. > In my > opinion, it does Python's reputation harm to make such a statement. > Conservative users (probably the vast majority of Python users) > don't like to hear that software they are considering using is > nearing the end of its life. What does it gain us to announce that > the 2.x branch is dead aside from bugfixes? > > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. > > Said another way, it's okay for the Python developers to decide to > abandon 2.x and put their efforts into 3.x. It's not okay for them > to prevent others from continuing to work on 2.x or to somehow make > 2.x look worse so 3.x looks better. Python 3 needs to stand on its > own terms and I'm confident it can. > I don't think ending the 2.x series at 2.7 makes it look bad compared to 3.2; it's simply the end of a development line like any other software project. I suspect 2.7 will have a protracted bugfix window because so much code runs on 2.x exclusively at the moment. And if core developers want to continue to backport fixes past two years from release they can (or however long we decide to officially support 2.7). No one is saying people still can't work on the code, just that python-dev as an entity is not going to focus its effort on the 2.x series anymore and people should not rely upon us to continue to focus new development efforts in that series. If there are core developers who want to continue to do bugfix releases then that's fine and I am happy to flag patches as needing backports and let other's do the work after the standard two year maintenance cycle, but I know I do not want to be held accountable as a core developer to keep the 2.x going indefinitely. Maintaining four branches is enough of a reason in my book to not keep the 2.x series going. If there really is an outcry on this we can re-visit the issue, but as of right now we need to move forward at some point and 2.7 seems like that good point. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 10 21:23:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Jan 2010 06:23:27 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A373F.9050601@gmail.com> Neil Schemenauer wrote: > In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. The announcement is precisely to avoid the situation where people commit new features to the 2.x main line of development (after the 2.7 maintenance branch is created) in the expectation that they will be released as part of a hypothetical 2.8 release. Whether that info needs to be in each and every 2.7 announcement... probably not. It isn't really info for users of Python, just for developers of Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jan 10 21:25:29 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:25:29 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit Message-ID: Michael has given me the hg transition/stdlib time slot at the language summit this year. In regards to that I plan to lead a discussion on: * where we are at w/ the Hg transition (Dirkjan should be there and I did a blog post on this topic recently: http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) * argparse (PEP 389) * brief mention on still wanting to break out the stdlib from CPython * an official policy on extension modules? (i.e. must have a pure Python implementation which can mean a ctypes implementation unless given an explicit waiver) That's everything from a stdlib perspective. I have half-baked ideas, but nothing concrete enough to bring up unless people really want to discuss stuff like how to potentially standardize module deprecation warnings, etc. In terms of me personally, I do plan to bring up at some point during the summit these points which don't squarely fit during my time slot: * an official unofficial policy on how new proposed features should come to us (i.e. working code to python-ideas with a shell of a PEP that includes abstract and motivation -> hashed out PEP to python-dev -> pronouncement) * any changes needed to the issue tracker to help with the workflow? (stage field seems like a failed experiment and we now have several effective triage people who can help w/ guiding changes) If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Sun Jan 10 22:52:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 10 Jan 2010 22:52:00 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) Unfortunately my flight doesn't land until about the time the Mercurial slot ends, so while I'll be there later on that afternoon for sprinting or questions or anything, I won't be at the actual Mercurial talk at the summit. Cheers, Dirkjan From fuzzyman at voidspace.org.uk Sun Jan 10 23:27:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 10 Jan 2010 22:27:58 +0000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4A546E.8010808@voidspace.org.uk> On 10/01/2010 21:52, Dirkjan Ochtman wrote: > On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > >> Michael has given me the hg transition/stdlib time slot at the language >> summit this year. In regards to that I plan to lead a discussion on: >> * where we are at w/ the Hg transition (Dirkjan should be there and I did a >> blog post on this topic recently: >> http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) >> > Unfortunately my flight doesn't land until about the time the > Mercurial slot ends, so while I'll be there later on that afternoon > for sprinting or questions or anything, I won't be at the actual > Mercurial talk at the summit. > > We will probably be in 'open discussion' when you arrive so it will still be good to hear from you on the topic and there maybe questions that can't be answered until you arrive... :-) Michael > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Mon Jan 11 00:02:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:02:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A5C69.7090007@v.loewis.de> > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. I think it's more difficult than that. It also takes developer effort to *commit* to the trunk. So if the trunk continued to live, would it still be ok if I closed a 2.x feature request as "won't fix", or only committed the new feature to the 3.x branch? As for old branches: they *don't* live in the way you claim (i.e. remain open with changes potentially just not released). Instead, at some point, they are frozen to bug-fix only, then to security-fix only, and then to no change at all. Regards, Martin From martin at v.loewis.de Mon Jan 11 00:07:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:07:58 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A373F.9050601@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> Message-ID: <4B4A5DCE.3070909@v.loewis.de> > The announcement is precisely to avoid the situation where people commit > new features to the 2.x main line of development (after the 2.7 > maintenance branch is created) in the expectation that they will be > released as part of a hypothetical 2.8 release. > > Whether that info needs to be in each and every 2.7 announcement... > probably not. It isn't really info for users of Python, just for > developers of Python. I think the announcement is indeed also for the users, and I would prefer it to stay in the release announcements, so that people will know for sure (and speak up) before the 2.7 release happens. As for decisions: I don't think there was an official BDFL pronouncent, but I recall Guido posting a message close to that, proposing that 2.7 will be a release that will see bug fix releases for an indefinite period of time (where indefinite != infinite). This was shortly after him proposing that perhaps we shouldn't make a 2.7 release at all, and stop at 2.6. As for such a decision giving a bad light on Python: I don't think that will be the case. Instead, I hear many users surprised for how long we have been maintaining to parallel versions - "that must have taken a lot of effort". So everybody will likely understand that enough is enough. Regards, Martin From benjamin at python.org Mon Jan 11 01:07:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 18:07:35 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Consider this program: class Descr(object): def __init__(self, name): self.name = name def __set__(self, instance, what): instance.__dict__[self.name] = what class X(object): attr = Descr("attr") x = X() print(x.attr) x.attr = 42 print(x.attr) It gives in output: <__main__.Descr object at 0x7fe1c9b28150> 42 The documentation [1] says that Descr is a data descriptor because it defines the __set__ method. It also states that data descriptors always override the value in the instance dictionary. So, the second line should also be the descriptor object according to the documentation. My question is: Is this a doc bug or a implementation bug? If the former, it will be the description of a data descriptor much less consistent, since it will require that a __get__ method be present, too. If the latter, the fix may break some programs relying on the ability to "cache" a value in the instance dictionary. [1] http://docs.python.org/reference/datamodel#invoking-descriptors -- Regards, Benjamin From amauryfa at gmail.com Mon Jan 11 01:51:09 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Jan 2010 01:51:09 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: Hi, 2010/1/11 Benjamin Peterson : > Consider this program: > > class Descr(object): > ? ?def __init__(self, name): > ? ? ? ?self.name = name > ? ?def __set__(self, instance, what): > ? ? ? ?instance.__dict__[self.name] = what > > class X(object): > ? ?attr = Descr("attr") > > x = X() > print(x.attr) > x.attr = 42 > print(x.attr) > > It gives in output: > > <__main__.Descr object at 0x7fe1c9b28150> > 42 > > The documentation [1] says that Descr is a data descriptor because it > defines the __set__ method. It also states that data descriptors > always override the value in the instance dictionary. So, the second > line should also be the descriptor object according to the > documentation. > > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors Quoting the documentation: """Normally, data descriptors define both __get__() and __set__(), while non-data descriptors have just the __get__() method. """ Your example is neither a data descriptor nor a non-data descriptor... The thing that worries me a bit is the "x.attr" returning the Descr object. Descriptors should remain at the class level, and instance should only see values. I'd prefer an AttributeError in this case. -- Amaury Forgeot d'Arc From benjamin at python.org Mon Jan 11 02:00:32 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 19:00:32 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <1afaf6161001101700u21006708l2ef62bfa257e4ce7@mail.gmail.com> 2010/1/10 Amaury Forgeot d'Arc : > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... See the footnote: http://docs.python.org/reference/datamodel#id7 > > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. Far too late for that, I'm afraid. -- Regards, Benjamin From nas at arctrix.com Mon Jan 11 02:44:57 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:44:57 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <20100111014457.GA5289@arctrix.com> On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > I don't think ending the 2.x series at 2.7 makes it look bad > compared to 3.2; it's simply the end of a development line like > any other software project. I suspect 2.7 will have a protracted > bugfix window because so much code runs on 2.x exclusively at the > moment. I would guess over 99% of all Python code written doesn't run on Python 3. Given that, I think it is premature to close the door on new major versions of Python 2.x. Also, we as a project should be careful not to present the image that Python 2.x will not be supported in the future. > If there really is an outcry on this we can re-visit the issue, > but as of right now we need to move forward at some point and 2.7 > seems like that good point. I think that's bad PR. If I had a successful product, I would not announce its end of life just to see how many customers scream and then decide if I should devote more resources to continue maintaining it. IMHO, the release notes should say something like: After the Python 2.7 release, the focus of Python development will be on Python 3. There will continue to be maintainance releases of Python 2.x. trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil From tjreedy at udel.edu Mon Jan 11 03:26:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Jan 2010 21:26:34 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On 1/10/2010 8:44 PM, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: >> I don't think ending the 2.x series at 2.7 makes it look bad >> compared to 3.2; it's simply the end of a development line like >> any other software project. I suspect 2.7 will have a protracted >> bugfix window because so much code runs on 2.x exclusively at the >> moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. If the removal of old features had been done in the 2.x series, as once planned (Guido originally proposed removing the old meaning of int / int in 2.5) the same more or less would be true of 2.7. It is past time for other old and now duplicated features to be removed also. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > >> If there really is an outcry on this we can re-visit the issue, >> but as of right now we need to move forward at some point and 2.7 >> seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. Python is not being ended, but upgraded (with bloating duplications removed). Think of 3.1 as 2.8 with a new name. tjr From lrekucki at gmail.com Mon Jan 11 04:26:48 2010 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Mon, 11 Jan 2010 04:26:48 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: > Date: Mon, 11 Jan 2010 01:51:09 +0100 > From: "Amaury Forgeot d'Arc" > To: Benjamin Peterson > Cc: Python Dev > Subject: Re: [Python-Dev] Data descriptor doc/implementation > ? ? ? ?inconsistency > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > 2010/1/11 Benjamin Peterson : >> Consider this program: >> >> class Descr(object): >> ? ?def __init__(self, name): >> ? ? ? ?self.name = name >> ? ?def __set__(self, instance, what): >> ? ? ? ?instance.__dict__[self.name] = what >> >> class X(object): >> ? ?attr = Descr("attr") >> >> x = X() >> print(x.attr) >> x.attr = 42 >> print(x.attr) >> >> It gives in output: >> >> <__main__.Descr object at 0x7fe1c9b28150> >> 42 >> >> The documentation [1] says that Descr is a data descriptor because it >> defines the __set__ method. It also states that data descriptors >> always override the value in the instance dictionary. So, the second >> line should also be the descriptor object according to the >> documentation. >> >> My question is: Is this a doc bug or a implementation bug? If the >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors > > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... Actually, there is this footnote [1]: """ A descriptor can define any combination of __get__(), __set__() and __delete__(). If it does not define __get__(), then accessing the attribute even on an instance will return the descriptor object itself. If the descriptor defines __set__() and/or __delete__(), it is a data descriptor; if it defines neither, it is a non-data descriptor. """ Which would mean Descr is actually a data descriptor without a __get__(), so x.attr should always return the descriptor object itself (at least in the docs). > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. > > -- > Amaury Forgeot d'Arc [1] http://docs.python.org/reference/datamodel#id7 -- Lukasz Rekucki From brett at python.org Mon Jan 11 04:46:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 19:46:04 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > > I don't think ending the 2.x series at 2.7 makes it look bad > > compared to 3.2; it's simply the end of a development line like > > any other software project. I suspect 2.7 will have a protracted > > bugfix window because so much code runs on 2.x exclusively at the > > moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Well yeah; Python 3.0 is just over a year old with 3.1 -- the first robust 3.x release - is about six months old. From the beginning uptake was expected to take years, not months, so saying that 3.x is not popular enough seems premature. > Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > No one has said bugfixes will cease. > > > If there really is an outcry on this we can re-visit the issue, > > but as of right now we need to move forward at some point and 2.7 > > seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. > > I never said that I wanted to make this announcement specifically to provoke outcries. I said if there happened to be outcries we could possibly visit the issue again. Basically I was being considerate and trying to leave the door open to discuss things in the future even though I don't see the situation changing. > IMHO, the release notes should say something like: > > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. > No because that suggests new features will be coming to 2.x which is not going to happen. If you want to say there will be continual bugfix releases for 2.7 as is par the course for Python and that the number of bugfix releases might be more than normal then I am okay with that. > > > trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil > That came and went already a couple months ago when we discussed stopping at 2.6 instead of 2.7 (see the various threads at http://mail.python.org/pipermail/python-dev/2009-November/thread.html). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 05:05:07 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:05:07 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <20100111040507.GA7200@arctrix.com> On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > After the Python 2.7 release, the focus of Python development > > will be on Python 3. There will continue to be maintainance > > releases of Python 2.x. > > No because that suggests new features will be coming to 2.x which is not > going to happen. If you want to say there will be continual bugfix releases > for 2.7 as is par the course for Python and that the number of bugfix > releases might be more than normal then I am okay with that. Are you are saying that if someone steps up to merge the Unladen Swallow features into a 2.8 release and someone also steps up to cut the release that they will be prevented from doing so? Also, are you presuming to channel the BDFL or was this dicussed somewhere other than python-dev? Maybe I'm overreacting but I get the feeling that the larger and less active segment of the Python develpment team hasn't been involved in these decisions. Best regards, Neil From nas at arctrix.com Mon Jan 11 05:17:54 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:17:54 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5C69.7090007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> Message-ID: <20100111041754.GB7200@arctrix.com> On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: > [...] would it still be ok if I closed a 2.x feature request as > "won't fix", or only committed the new feature to the 3.x branch? Yes. Non-bugfix development on 2.x would optional (i.e. done by people who want to spend the time). Since language changes are now out (an initiative I completely support), the majority of non-bugfix changes would be optimizations and platform porting. Regards, Neil From brett at python.org Mon Jan 11 06:06:15 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 21:06:15 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111040507.GA7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 20:05, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > > After the Python 2.7 release, the focus of Python development > > > will be on Python 3. There will continue to be maintainance > > > releases of Python 2.x. > > > > No because that suggests new features will be coming to 2.x which is not > > going to happen. If you want to say there will be continual bugfix > releases > > for 2.7 as is par the course for Python and that the number of bugfix > > releases might be more than normal then I am okay with that. > > Are you are saying that if someone steps up to merge the Unladen > Swallow features into a 2.8 release and someone also steps up to cut > the release that they will be prevented from doing so? > > I honestly don't know, but it's a possibility just like any other new feature request. If people start taking the carrots we have added to 3.x and backporting them to keep the 2.x series alive you are essentially making the 3.x DOA by negating its benefits which I personally don't agree with. > Also, are you presuming to channel the BDFL or was this dicussed > somewhere other than python-dev? This was discussed; see the November 2009 threads. Guido actually suggested ending the 2.x branch at 2.6 until people spoke up about the amount of stuff already backported to 2.7 from 3.x because it was being assumed to be the end of the series to warrant keeping it to help transition to 2.7. > Maybe I'm overreacting but I get > the feeling that the larger and less active segment of the Python > develpment team hasn't been involved in these decisions. > This all came up in November from the 3rd through the 6th (four days) over a ton of email traffic. This was not a snap decision but a heated discussion where even Guido participated that culminated with the decision to stop at 2.7 for the 2.x series; this was not a smoked-filled room decision. I'm sorry if people missed it when they weren't looking, but python-dev is supposed to be the place to watch for this kind of stuff. I don't know how else to bring this stuff to people's attention short of also on python-committers, but developers are basically expected to be on both lists so I don't see where anyone did anything wrong in terms of informing developers. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 06:27:13 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 23:27:13 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: <20100111052713.GA8211@arctrix.com> On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > If people start taking the carrots we have added to 3.x and > backporting them to keep the 2.x series alive you are essentially > making the 3.x DOA by negating its benefits which I personally > don't agree with. I think we have got to the heart of our disagreement. Assume that some superhuman takes all the backwards compatible goodies from 3.x and merges them into 2.x. If that really would kill off Python 3 then I would proclaim it a project that deserved to die. Why should the backwards incompatible changes be endured if they cannot justify their existance by the benefit they provide? I guess I have more confidence in Python 3 than you do. I don't see why Python 2.x needs to be artificially limited so that Python 3 can benefit. Regards, Neil From brett at python.org Mon Jan 11 07:30:49 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 22:30:49 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 21:27, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. > > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. If that really would kill off Python 3 > then I would proclaim it a project that deserved to die. Why should > the backwards incompatible changes be endured if they cannot justify > their existance by the benefit they provide? > > When I said "carrots" I meant everything that makes Python 3 the best version of Python, including the backward-incompatible changes. I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. > I don't view it as artificial but simply a focusing of the development team on what has been determined as the future of Python by Guido and letting go of the past. IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying "Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out". That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Jan 11 08:08:14 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 08:08:14 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <319e029f1001102308p5de87cber2131f3aec1abc9f0@mail.gmail.com> On Mon, Jan 11, 2010 at 06:27, Neil Schemenauer wrote: > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. The superhumans of core developers pretty much already have. That's pretty much 2.7 you are talking about. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From chrism at plope.com Mon Jan 11 08:27:03 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 11 Jan 2010 02:27:03 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4AD2C7.1050703@plope.com> Brett Cannon wrote: > IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev > saying "Python 3 is the future, but we are keeping the old Python 2.x > around because we don't have *that* much faith in the future we have > laid out". That's poison to Python 3 by showing a lack of confidence in > the direction that the BDFL and python-dev as a group has chosen. Now I > could be wrong and there could actually be a large number of active > contributors who want to keep the 2.x series going, but based on the > discussion that occurred the last time this came up I believe the guys > who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort "Python 2". - C From martin at v.loewis.de Mon Jan 11 09:06:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:06:16 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: <4B4ADBF8.3030803@v.loewis.de> Neil Schemenauer wrote: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). I think *that* would give a very bad impression of Python. Depending whom you deal with, the new feature you want may or may not get added to the code base. Contributors would feel even more stranded than they do now, where it may take several years to get a patch reviewed, as you then could submit a patch, and pray that a comitter reviews it who believes in future 2.x releases. The point of setting policies is that it gives every user (contributors, committers, and "end-user" developers) a reliable foundation for expectations. Regards, Martin From arcriley at gmail.com Mon Jan 11 09:06:10 2010 From: arcriley at gmail.com (Arc Riley) Date: Mon, 11 Jan 2010 03:06:10 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AD2C7.1050703@plope.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4AD2C7.1050703@plope.com> Message-ID: after all these years, some people still run AmigaOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 11 09:11:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:11:25 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <4B4ADD2D.6070906@v.loewis.de> > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. Why that? It is a fact: 2.x will not be supported, in some future. Should we lie to users? > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. It shouldn't say that, because it wouldn't be true. Regards, Martin From martin at v.loewis.de Mon Jan 11 09:18:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:18:30 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4ADED6.5080207@v.loewis.de> > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. Because it takes too much time. Too much of my time, but apparently also too much of other people's time. Of course, the less active fraction of Python contributors may not notice, since they just chose to not contribute (which, of course, is fine). However, asking me to work twice as much as I want to on the project to keep two branches alive is just unfair. This has nothing to do with pushing 3.x, but all with managing available manpower and still providing quality software. Regards, Martin From regebro at gmail.com Mon Jan 11 09:42:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 09:42:51 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> This is what it says now: "Python 2.7 is scheduled to be the last major version in the 2.x series before it moves into 5 years of bugfix-only mode. " And during this discussion it has been noted that others than the core python team might pick up Python 2 and make releases. So maybe we can and this discussion by changing that line in future releases to be: "Python 2.7 is scheduled to be the last major version in the 2.x series released by the Python Software Foundation before it moves into 5 years of bugfix-only mode. " That doesn't exclude *others* making a Python 2.8 that could include all sorts of crazy features. :) This is mainly just to get the pointless discussion over with. The current Python core team don't want to make a 2.8, so that will not happen. Someone else might, but as Chris points out, they won't step up until they have to, which is probably at least two years from now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 10:06:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 10:06:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> Message-ID: <4B4AEA25.7010100@v.loewis.de> > "Python 2.7 is scheduled to be the last major version in the 2.x > series released by the Python Software Foundation before it moves into > 5 years of bugfix-only mode. " > > That doesn't exclude *others* making a Python 2.8 that could include > all sorts of crazy features. :) > > This is mainly just to get the pointless discussion over with. I know you are just looking for a compromise, but this shouldn't be it: the PSF has deliberately stayed out of the actual Python engineering, so the release that Benjamin makes is not done by the PSF (but by Benjamin and his contributors :-). I like the wording as it is (although I find the promise of five years of bug fix releases a bit too strong). Regards, Martin From regebro at gmail.com Mon Jan 11 10:19:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 10:19:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AEA25.7010100@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> Message-ID: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" wrote: > I know you are just looking for a compromise, but this shouldn't be it: > the PSF has deliberately stayed out of the actual Python engineering, > so the release that Benjamin makes is not done by the PSF (but by > Benjamin and his contributors :-). Hm. Yeah. That's right of course. I started with saying "official", but then I thought "official according to who?". :) So I changed it to mentioning PSF, but that doesn't work either. I guess the current writing as as good as it gets, unless we change "scheduled" to "expected" or something. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Mon Jan 11 10:42:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:42:05 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: Neil Schemenauer, 11.01.2010 05:17: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). Note that there's also the time it takes to make a release (usually involving at least one beta release), plus the possibility of introducing bugs while adding new features or even while fixing agreed bugs. All of this takes time in addition to the time people might want to invest in 'real' development on the 2.x trunk. Stefan From stephen at xemacs.org Mon Jan 11 10:59:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Jan 2010 18:59:57 +0900 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <8763793qsi.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Schemenauer writes: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. Well, I think it's *worse* than that, and I don't think you really mean "DOA", anyway. (Feel free to correct me, of course.) The problem I see with backporting lots of stuff, and/or adding new features that aren't in 3.0, to 2.x is that it will make 2.x even cruftier, when it was already crufty enough that Guido (and almost all of python-dev) bit the bullet and said "backward compatibility is no excuse for keeping something in 3.0". That surely means that a lot of python-dev denizens will declare non-support 2.x for x > 7. It's not going to be the gradual migration we've seen over the past few months as active people start to spend more and more time on 3 vs. 2; it will be a watershed. Especially if these are new features merged from outside that the "small active segment" doesn't know anything about. From the users' point of view, that amounts to a *fork*, even if it's internal and "friendly". > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. Isn't that a bit ridiculous? I just don't see any evidence that anything like that is going to happen. Worse, if we *assume* it will happen, I don't see any way to assess whether (1) Python 3 goes belly up, (2) there's an effective fork confusing the users and draining the energy of python-dev, or (3) everybody goes "wow!" because it's so cool that everybody wants to keep maintaining an extra 3 branches indefinitely. My opinion is that given the clear direction the "small active segment" is going, telling the users anything but what Brett proposed is disinformation. > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. It's not for Python 3, which you, I, and I'm pretty sure Brett-in-his- heart-of-hearts agree can take care of itself because it is *better* than Python 2. It's for Python, and for the Python community. From walter at livinglogic.de Mon Jan 11 11:37:56 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 11:37:56 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001091438.43576.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> Message-ID: <4B4AFF84.7070206@livinglogic.de> On 09.01.10 14:38, Victor Stinner wrote: > Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : >>> Good idea, I choosed open(filename, encoding="BOM"). >> >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Why not, this is another solution to the point (2) (Check for a BOM while > reading or detect it before?). Which encoding would be used if there is not > BOM? UTF-8 sounds like a good choice. UTF-8 might be a good choice, are the failback could be specified in the encoding name, i.e. open("file.txt", encoding="BOM-UTF-8") falls back to UTF-8, if there's no BOM at the start. This could be implemented via a custom codec search function (see http://docs.python.org/library/codecs.html#codecs.register for more info). Servus, Walter From regebro at gmail.com Mon Jan 11 12:12:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:12:20 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4AFF84.7070206@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> Message-ID: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: > UTF-8 might be a good choice No, fallback if there is no BOM should be the local settings, just as fallback is today if you don't specify a codec. I mean, what if you want to look for a BOM but fall back to something else? How far will we go with encoding special information in the codecs names? codec='BOM else UTF-16 else locale'? :-) BOM is not a locale, and should not be a locale. Having a locale called BOM is wrong per se. It should either be default to look for a BOM when codec=None, or a special parameter. If none of these are desired, then we need a special function that takes a filename or file handle, and looks for a BOM and returns the codec found or None. But I find that much less natural and obvious than checking the BOM when codec=None. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Mon Jan 11 12:13:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:13:00 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <319e029f1001110313u1d839deodfe06a6c7ec423cf@mail.gmail.com> On Mon, Jan 11, 2010 at 12:12, Lennart Regebro wrote: > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. D'oh! I mean codec here obviously. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 13:29:04 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 13:29:04 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4913DF.7050801@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> Message-ID: <4B4B1990.90705@livinglogic.de> On 10.01.10 00:40, "Martin v. L?wis" wrote: >>> How does the requirement that it be implemented as a codec miss the >>> point? >> >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. > > Yes - however, Victor currently apparently *doesn't* want it to be the > default, but wants the user to specify encoding="BOM". If so, it isn't > the default, and it is easy to implement as a codec. > >>> FWIW, I agree with Walter that if it is provided through the encoding= >>> argument, it should be a codec. If it is built into the open function >>> (for whatever reason), it must be provided by some other parameter. >> >> Why not simply encoding=None? > > I don't mind. Please re-read Walter's message - it only said that > *if* this is activated through encoding="BOM", *then* it must be > a codec, and could be on PyPI. I don't think Walter was talking about > the case "it is not activated through encoding='BOM'" *at all*. However if this autodetection feature is useful in other cases (no matter how it's activated), it should be a codec, because as part of the open() function it isn't reusable. >> The default value should provide the most useful >> behaviour possible. Forcing users to choose between two different autodetection >> strategies (encoding=None and another one) is a little insane IMO. And encoding="mbcs" is a third option on Windows. > That wouldn't disturb me much. There are a lot of things in that area > that are a little insane, starting with Microsoft Windows :-) ;) Servus, Walter From barry at python.org Mon Jan 11 13:37:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 07:37:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5DCE.3070909@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: On Jan 10, 2010, at 6:07 PM, Martin v. L?wis wrote: > As for decisions: I don't think there was an official BDFL pronouncent, > but I recall Guido posting a message close to that, proposing that 2.7 > will be a release that will see bug fix releases for an indefinite > period of time (where indefinite != infinite). This was shortly after > him proposing that perhaps we shouldn't make a 2.7 release at all, and > stop at 2.6. IMO, the focus for Python 2.7 (and beyond) must be on helping people transition to Python 3. That should be the reason why a 2.7 is released and, what should dictate whether a 2.8 is necessary. Maybe everything people need (except manpower and round tuits) is already there. I was pleasantly surprised to find that Distribute supports automatically running 2to3 at install time for Python packages. This allowed me to "port" a recent (admittedly small) package to Python 3 by making it "python2.6 -3" clean and adding a couple of attributes to my setup.py[1]. I don't know how widely that feature's known, but I think it's *huge*, and exactly what we need to be doing. The more we can make it easy for people to port their Python 2 code to Python 3, and to support that with one code base, the quicker we'll get to a predominantly Python 3 world. So the question we should be asking is: what's missing from Python 2.7 to help with this transition? If we can't get it into 2.7, then why, and would pushing it back to 2.8 help at all? -Barry [1] modulo a bug in Distribute that caused doctest in separate files to not be used when running under Python 3. From solipsis at pitrou.net Mon Jan 11 13:39:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 13:39:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <1263213574.3327.0.camel@localhost> > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. It is reusable as part of io.TextIOWrapper, though. Regards Antoine. From regebro at gmail.com Mon Jan 11 13:45:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 13:45:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. But an autodetect feature is not a codec. Sure it should be reusable, but making it a codec seems to be a weird hack to me. And how would you reuse it if it was a codec? A reusable autodetect feature would be useable to detect what codec it is. A autodetect codec would not be useful for that, as it would simply just decode. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 14:21:15 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 14:21:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B25CB.5030003@livinglogic.de> On 11.01.10 13:45, Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: >> However if this autodetection feature is useful in other cases (no >> matter how it's activated), it should be a codec, because as part of the >> open() function it isn't reusable. > > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. I think we already had this discussion two years ago in the context of XML decoding ;): http://mail.python.org/pipermail/python-dev/2007-November/075138.html > And how would > you reuse it if it was a codec? A reusable autodetect feature would be > useable to detect what codec it is. A autodetect codec would not be > useful for that, as it would simply just decode. I have implemented an XML codec (as part of XIST: http://pypi.python.org/pypi/ll-xist), that can do that: >>> from ll import xml_codec >>> import codecs >>> c = codecs.getincrementaldecoder("xml")() >>> c.encoding >>> c.decode(">> c.encoding >>> c.decode(" version='1.0'") u'' >>> c.encoding >>> c.decode(" encoding='iso-8859-1'?>") u"" >>> c.encoding 'iso-8859-1' Servus, Walter From regebro at gmail.com Mon Jan 11 14:47:12 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 14:47:12 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B25CB.5030003@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B25CB.5030003@livinglogic.de> Message-ID: <319e029f1001110547n1d1c9831p34b0eac519d13f9d@mail.gmail.com> On Mon, Jan 11, 2010 at 14:21, Walter D?rwald wrote: > I think we already had this discussion two years ago in the context of > XML decoding ;): Yup. Ans Martins answer then is my answer now: "> So the code is good, if it is inside an XML parser, and it's bad if it > is inside a codec? Exactly so. This functionality just *isn't* a codec - there is no encoding. Instead, it is an algorithm for *detecting* an encoding." The conclusion was that a method do autodetect encodings would be good. I think the same conclusion applies here. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 18:16:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:16:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B5CF5.50806@v.loewis.de> > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. Well, the existing UTF-16 codec also is an autodetect feature (to detect the endianness), and I don't consider it a weird hack. Regards, Martin From regebro at gmail.com Mon Jan 11 18:27:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:27:01 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B5CF5.50806@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> Message-ID: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> On Mon, Jan 11, 2010 at 18:16, "Martin v. L?wis" wrote: >> But an autodetect feature is not a codec. Sure it should be reusable, >> but making it a codec seems to be ?a weird hack to me. > > Well, the existing UTF-16 codec also is an autodetect feature (to > detect the endianness), and I don't consider it a weird hack. So the BOM codec should raise a UnicodeDecodeError if there is no BOM? Because that's what it would have to do, in that case, because it can't fall back on anything, it has to handle and implement all encodings that have a BOM. And is it then actually very useful? You would have to do a try/except first with encoding='BOM' and then encoding=None to get the fallback to the standard. I must say that I find this whole thing pretty obvious. 'BOM' is not an encoding. Either there should be a method to get the encoding from the BOM, returning None of there isn't one, or open() should look at the BOM when you pass in encoding=None. Or both. That covers all usecases, is easy and obvious. Either open(file=foo, encoding=None) or open(file, encoding=encoding_from_bom(file)) I can't see that open(file, encoding='BOM') has any benefit over this, covers any extra usecase and is clearer in any way. Instead it adds something confusing: An encoding that isn't an encoding. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From python at mrabarnett.plus.com Mon Jan 11 18:35:35 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:35:35 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <4B4B6167.40606@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: >> UTF-8 might be a good choice > > No, fallback if there is no BOM should be the local settings, just as > fallback is today if you don't specify a codec. > I mean, what if you want to look for a BOM but fall back to something > else? How far will we go with encoding special information in the > codecs names? codec='BOM else UTF-16 else locale'? :-) > > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. It should either be default to look for a > BOM when codec=None, or a special parameter. If none of these are > desired, then we need a special function that takes a filename or file > handle, and looks for a BOM and returns the codec found or None. But > I find that much less natural and obvious than checking the BOM when codec=None. > Or pass a function that accepts a byte stream or the first few bytes and returns the encoding and any unused bytes (because the byte stream might not be seekable)? def guess_encoding(byte_stream): data = byte_stream.read(2) if data == b"\xFE\xFF": return "UTF-16BE", b"" return "UTF-8", data text_file = open(filename, encoding=guess_encoding) ... From python at mrabarnett.plus.com Mon Jan 11 18:46:30 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:46:30 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> Message-ID: <4B4B63F6.1020309@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" > wrote: >> I know you are just looking for a compromise, but this shouldn't be >> it: the PSF has deliberately stayed out of the actual Python >> engineering, so the release that Benjamin makes is not done by the >> PSF (but by Benjamin and his contributors :-). > > Hm. Yeah. That's right of course. I started with saying "official", > but then I thought "official according to who?". :) "Official" is whatever the BDFL says it is! :-) > So I changed it to mentioning PSF, but that doesn't work either. I > guess the current writing as as good as it gets, unless we change > "scheduled" to "expected" or something. > From martin at v.loewis.de Mon Jan 11 18:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:59:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> Message-ID: <4B4B66EC.7000203@v.loewis.de> > I must say that I find this whole thing pretty obvious. 'BOM' is not > an encoding. That I certainly agree with. > That covers all usecases, is easy and obvious. Either open(file=foo, > encoding=None) or open(file, encoding=encoding_from_bom(file)) > > I can't see that open(file, encoding='BOM') has any benefit over this, Well, it would have the advantage that Walter pointed out: you can implement it independent of the open() implementation, and even provide it in older versions of Python. Regards, Martin From regebro at gmail.com Mon Jan 11 18:59:52 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B63F6.1020309@mrabarnett.plus.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> Message-ID: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> On Mon, Jan 11, 2010 at 18:46, MRAB wrote: > "Official" is whatever the BDFL says it is! :-) Heh, right. So, it should say "Guido wants 2.7 to be the last main version of Python 2, so it probably will be. We promise to release bugfixes it for, like, ages". No need to be needlessly formal. :-D -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From olemis at gmail.com Mon Jan 11 19:58:01 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 13:58:01 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> [...] > I had similar issues too (please read below ;o) ... On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > About guessing the encoding, I experienced this issue while I was developing a Trac plugin. What I was doing is as follows : - I guessed the MIME type + charset encoding using Trac MIME API (it was a CSV file encoded using UTF-16) - I read the file using `open` - Then wrapped the file using `codecs.EncodedFile` - Then used `csv.reader` ... and still get the BOM in the first value of the first row in the CSV file. {{{ #!python >>> mimetype 'utf-16-le' >>> ef = EncodedFile(f, 'utf-8', mimetype) }}} IMO I think I am +1 for leaving `open` just like it is, and use module `codecs` to deal with encodings, but I am strongly -1 for returning the BOM while using `EncodedFile` (mainly because encoding is explicitly supplied in ;o) > --Guido > CMIIW anyway ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mal at egenix.com Mon Jan 11 21:44:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Jan 2010 21:44:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> Message-ID: <4B4B8DB2.1060102@egenix.com> Olemis Lang wrote: >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>> BOM should be "ignored". >>> > [...] >> > > I had similar issues too (please read below ;o) ... > > On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> > > About guessing the encoding, I experienced this issue while I was > developing a Trac plugin. What I was doing is as follows : > > - I guessed the MIME type + charset encoding using Trac MIME API (it > was a CSV file encoded using UTF-16) > - I read the file using `open` > - Then wrapped the file using `codecs.EncodedFile` > - Then used `csv.reader` > > ... and still get the BOM in the first value of the first row in the CSV file. You didn't say, but I presume that the charset guessing logic returned either 'utf-16-le' or 'utf-16-be' - those encodings don't remove the leading BOM. The 'utf-16' codec will remove the BOM. > {{{ > #!python > >>>> mimetype > 'utf-16-le' >>>> ef = EncodedFile(f, 'utf-8', mimetype) > }}} Same here: the UTF-8 codec will not remove the BOM, you have to use the 'utf-8-sig' codec for that. > IMO I think I am +1 for leaving `open` just like it is, and use module > `codecs` to deal with encodings, but I am strongly -1 for returning > the BOM while using `EncodedFile` (mainly because encoding is > explicitly supplied in ;o) Note that EncodedFile() doesn't do any fancy BOM detection or filtering. This is the job of the codecs. Also note that BOM removal is only valid at the beginning of a file. All subsequent BOM-bytes have to be read as-is (they map to a zero-width non-breaking space) - without removing them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 11 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Jan 11 22:12:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 07:12:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <4B4B9447.4060101@gmail.com> Benjamin Peterson wrote: > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors I would call it a documentation bug: by leaving out the __get__ you're telling Python to override *just* the setting of the attribute, while still using the normal lookup process for retrieving the attribute (i.e. allowing the attribute to be shadowed in the instance dictionary). Adding a "print('Descriptor Invoked')" to the __set__ method in your example: >>> x = X() >>> print(x.attr) <__main__.Descr object at 0x7f283b51ac50> >>> x.attr = 42 Descriptor invoked >>> print(x.attr) 42 >>> x.attr = 6*9 Descriptor invoked >>> print(x.attr) 54 Note that the behaviour here is still different from that of a data descriptor: with a data descriptor, once it gets shadowed in the instance dictionary, the descriptor is ignored *completely*. The only way to get the descriptor involved again is to eliminate the shadowing. The non-data descriptor with only __set__ is just choosing not to override the attribute lookup process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Mon Jan 11 22:29:38 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 16:29:38 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B8DB2.1060102@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> <4B4B8DB2.1060102@egenix.com> Message-ID: <24ea26601001111329i7f609aa1i9f5db7eb61f1fae7@mail.gmail.com> Probably one part of this is OT , but I think it could complement the discussion ;o) On Mon, Jan 11, 2010 at 3:44 PM, M.-A. Lemburg wrote: > Olemis Lang wrote: >>> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >>> wrote: >>>> Hi, >>>> >>>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>>> BOM should be "ignored". >>>> >> [...] >>> >> >> I had similar issues too (please read below ;o) ... >> >> On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >>> >> >> About guessing the encoding, I experienced this issue while I was >> developing a Trac plugin. What I was doing is as follows : >> >> - I guessed the MIME type + charset encoding using Trac MIME API (it >> was a CSV file encoded using UTF-16) >> - I read the file using `open` >> - Then wrapped the file using `codecs.EncodedFile` >> - Then used `csv.reader` >> >> ... and still get the BOM in the first value of the first row in the CSV file. > > You didn't say, but I presume that the charset guessing logic > returned either 'utf-16-le' or 'utf-16-be' Yes. In fact they return the full mimetype 'text/csv; charset=utf-16-le' ... ;o) > - those encodings don't > remove the leading BOM. ... and they should ? > The 'utf-16' codec will remove the BOM. > In this particular case there's nothing I can do, I have to process whatever charset is detected in the input ;o) >> {{{ >> #!python >> >>>>> mimetype >> 'utf-16-le' >>>>> ef = EncodedFile(f, 'utf-8', mimetype) >> }}} > > Same here: the UTF-8 codec will not remove the BOM, you have > to use the 'utf-8-sig' codec for that. > >> IMO I think I am +1 for leaving `open` just like it is, and use module >> `codecs` to deal with encodings, but I am strongly -1 for returning >> the BOM while using `EncodedFile` (mainly because encoding is >> explicitly supplied in ;o) > > Note that EncodedFile() doesn't do any fancy BOM detection or > filtering. ... directly. > This is the job of the codecs. > OK ... to come back to the scope of the subject, in the general case, I think that BOM (if any) should be handled by codecs, and therefore indirectly by EncodedFile . If that's a explicit way of working with encodings I'd prefer to use that wrapper explicitly in order to (encode | decode) the file and let the codec detect whether there's a BOM or not and ?adjust? `tell`, `read` and everything else in that wrapper (instead of `open`). > Also note that BOM removal is only valid at the beginning of > a file. All subsequent BOM-bytes have to be read as-is (they > map to a zero-width non-breaking space) - without removing them. > ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Test cases for custom query (i.e report 9) ... PASS (1.0.0) - http://simelo.hg.sourceforge.net/hgweb/simelo/trac-gviz/rev/d276011e7297 From martin at v.loewis.de Mon Jan 11 22:42:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:42:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: <4B4B9B55.1010709@v.loewis.de> > So the question we should be asking is: what's missing from Python > 2.7 to help with this transition? Wrt. the distribute feature you've noticed: Python 3.0 already supports that in distutils. So from day one of Python 3.x, you could have used that approach for porting Python code. I had been promoting it ever since, but it's taking up only slowly, partially because it has competition in the approaches "burn the bridges" (i.e. convert to 3.x once, and then have two code bases, or abandon 2.x), and "avoid 2to3" (i.e. try to write code that works in both 2.x and 3.x unmodified). > If we can't get it into 2.7, then > why, and would pushing it back to 2.8 help at all? I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: a) telling people that they have to move to 2.6 first actually hurts migration, instead of helping, because it implies to them that they have to drop old versions (e.g. 2.3.) - just because they had *always* dropped old versions before supporting new ones. b) IMO, people also don't gain much by first migrating to 2.6. In principle, it gives them the opportunity to get py3k warnings. However, I haven't heard a single positive report where these warnings have actually helped people in porting. Yours is the first report saying that you followed the official guideline, but you didn't state whether doing so actually helped (or whether you just ported to 2.6 because the guideline told you to). c) whatever 2.7 does (except perhaps for the warnings), it won't be useful to applications, because they couldn't use it, anyway: they need to support 2.4 and 2.5, and it won't have any of the gimmicks people come up with for 2.7. Adding gimmicks to 2.7 might actually hurt porting: people may have to special-case 2.7 because their work-arounds for older versions may break in 2.7 (e.g. testing whether a name is *not* defined, when it becomes defined in 2.7), plus it gives them an incentive to not port yet until they can drop support for anything before 2.7. Inherently, 2.8 can't improve on that. Regards, Martin From martin at v.loewis.de Mon Jan 11 22:44:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:44:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> Message-ID: <4B4B9BB8.3070505@v.loewis.de> > So, it should say "Guido wants 2.7 to be the last main version of > Python 2, so it probably will be. We promise to release bugfixes it > for, like, ages". > > No need to be needlessly formal. :-D That summarizes my understanding of what Guido said fairly well :-) +1 for putting it into the announcements. Regards, Martin From fuzzyman at voidspace.org.uk Mon Jan 11 23:11:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 11 Jan 2010 22:11:44 +0000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4B9447.4060101@gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> Message-ID: <4B4BA220.20701@voidspace.org.uk> On 11/01/2010 21:12, Nick Coghlan wrote: > Benjamin Peterson wrote: > > My question is: Is this a doc bug or a implementation bug? If the > >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors >> > [snip...] > > Note that the behaviour here is still different from that of a data > descriptor: with a data descriptor, once it gets shadowed in the > instance dictionary, the descriptor is ignored *completely*. The only > way to get the descriptor involved again is to eliminate the shadowing. > The non-data descriptor with only __set__ is just choosing not to > override the attribute lookup process. > > Does that mean we need a third class of descriptors that are neither data descriptors nor non-data descriptors? What should we call them: really-not-data-descriptors? All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Mon Jan 11 23:55:01 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Jan 2010 14:55:01 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9BB8.3070505@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: +1 from me. (With the caveat that "time will tell" is definitely the ultimate answer. Plans may change unexpectedly, even though we don't expect them to.) PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not helpful. But I trust he has ported a lot more code to 3.x than I have personally. Are there other experiences? --Guido On Mon, Jan 11, 2010 at 1:44 PM, "Martin v. L?wis" wrote: >> So, it should say "Guido wants 2.7 to be the last main version of >> Python 2, so it probably will be. We promise to release bugfixes it >> for, like, ages". >> >> No need to be needlessly formal. :-D > > That summarizes my understanding of what Guido said fairly well :-) > > +1 for putting it into the announcements. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Jan 12 00:16:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 00:16:47 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <4B4BB15F.5020807@v.loewis.de> Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. That's really because I haven't even attempted to use it. Instead, the software would fall flat on its own when running the test suite in 3.x. IOW, I don't need 2.6 to tell me what might break when I can see 3.x actually breaking. Regards, Martin From david.lyon at preisshare.net Tue Jan 12 00:22:42 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:22:42 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Hi Martin, > Of course, the less active fraction of Python contributors may not > notice, since they just chose to not contribute (which, of course, > is fine). However, asking me to work twice as much as I want to > on the project to keep two branches alive is just unfair. Totally true. Actually as an end-developer I'd say that python 2.x series from a programming perspective is quite good. It doesn't need the addition of a lot of new features imho. So for me, I think too much time spent there would be not yield great benefits. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Well, we all know your work is super quality. :-) That's not being contested. However, Quality can be measured different ways and it can be assessed in different ways. Quality itself is a subjective thing. The point I'm only making is that if a piece of software doesn't have "new" things added over time, then users can get a reverse impression of a lack of quality. We've all seen where 'internal' quality can increase and user perceptions can decrease. It could be things like improved graphics and things readily apparent to the user. At the moment, I would say that the "internal" quality of the python 2.x series is super high. "external" quality issues such as the packaging dilemma give the user the opposite quality experience. But people are working on that as best they can elsewhere. I'll leave it at that. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Python 3.x needs more carrots. >From an ordinary (perphaps ignorant) user perspective there is nothing. Yes, we know if we actually will start programming then we will like it more. But my wishes to Santa Claus would be allow the free flow of PEPs for Python 3 packaging. Even encourage it. As an end developer, here's what I'd like from Santa in 2010 to get me to swap to python 3: * get all the packages on pypi tested for python 3 * put a web based package manager in python 3. This would perhaps be based around PIP (keep many people happy) and would look much like the built in web-console that you get with a $200 router. * Incorporate SCM based end-user software installs by adding to python3-stdlib the SCM support packages (cvs,bzr,svn,hg). This would *really* help in the scientific community. * put a web interface on distutils also so that we don't have to use a command line interface. (I want a pic of a smiley girl to great me when I build something - "Are you ready to build now Honey?"). ok - I joke. But the point is made. So, ok, maybe these things aren't about 'code' quality. But rather user experience. Things like these do count also as "quality" via the technical term "perception of quality". If the PEP process is as unblocked as the documentation implies, implying that anybody can contribute to Python 3. Then there shouldn't be any issue. David From solipsis at pitrou.net Tue Jan 12 00:37:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 23:37:47 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon preisshare.net> writes: > > > This has nothing to do with pushing 3.x, but all with managing > > available manpower and still providing quality software. > > Python 3.x needs more carrots. As someone who experiences the difference almost every day, I can say 3.x definitely has enough carrots for me. The simple fact that I will be able to raise exceptions with non-ASCII error messages without having to workaround a hopelessly broken conversion/reporting logic is a huge improvement. And that's only a small part of the picture. Regards Antoine. From janssen at parc.com Tue Jan 12 00:47:55 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 11 Jan 2010 15:47:55 PST Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <17215.1263253675@parc.com> > David Lyon preisshare.net> writes: > > > > > This has nothing to do with pushing 3.x, but all with managing > > > available manpower and still providing quality software. > > > > Python 3.x needs more carrots. I'd be happy to move UpLib to Python 3, when the various packages that I need are also on Python 3. And that depresses me to think about. I need Medusa ReportLab PyLucene Email Vobject Mutagen Pyglet Hachoir Win32 I'm on the mailing lists for a lot of these, and the only one that I know is thinking about Python 3 is Email. I'd expect Win32 and PIL to also be thinking about it, but I haven't heard anything. Bill From david.lyon at preisshare.net Tue Jan 12 00:47:51 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:47:51 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <1220.218.214.45.58.1263253671.squirrel@syd-srv02.ezyreg.com> Antoine writes: > As someone who experiences the difference almost every day, I can say 3.x > definitely has enough carrots for me. The simple fact that I will be able > to > raise exceptions with non-ASCII error messages without having to > workaround a > hopelessly broken conversion/reporting logic is a huge improvement. And > that's > only a small part of the picture. In no way could I disagree with you. Ascii works fine for us here - but I guess we're lucky. Just keep pushing python 3 and have a nice day.. David From barry at python.org Tue Jan 12 01:11:28 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:11:28 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100111191128.39020d89@freewill> On Jan 11, 2010, at 10:42 PM, Martin v. L?wis wrote: >Wrt. the distribute feature you've noticed: Python 3.0 already supports >that in distutils. So from day one of Python 3.x, you could have used >that approach for porting Python code. I had been promoting it ever >since, but it's taking up only slowly, partially because it has >competition in the approaches "burn the bridges" (i.e. convert to 3.x >once, and then have two code bases, or abandon 2.x), and "avoid 2to3" >(i.e. try to write code that works in both 2.x and 3.x unmodified). Interesting. The only reason I never used it before is because I didn't know about it. Am I alone? Maybe I'm projecting my own preferences, but it seems to me that supporting both Python 2 and 3 from the same code base would be a very powerful way to enable a large amount of existing code to support Python 3. I'm certainly going to do this from now on with all the libraries I maintain. I don't have any cycles to write code twice and I can't abandon Python 2 yet. I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. As an example, the one library I've already ported used a metaclass. I don't see any way to specify that the metaclass should be used in a portable way. In Python 2.6 it's: class Foo: __metaclass__ = Meta and in Python 3 it's: class Foo(metaclass=Meta): 2to3 made that pain go away. >I've done a fair bit of 3.x porting, and I'm firmly convinced that >2.x can do nothing: > >a) telling people that they have to move to 2.6 first actually > hurts migration, instead of helping, because it implies to them > that they have to drop old versions (e.g. 2.3.) - just because > they had *always* dropped old versions before supporting new ones. Is it just an implication, or is it reality? I have yet to try to support older versions of Python and also support Python 3. (The one package I've done this with so far only supports Python 2.6.) >b) IMO, people also don't gain much by first migrating to 2.6. > In principle, it gives them the opportunity to get py3k warnings. > However, I haven't heard a single positive report where these > warnings have actually helped people in porting. Yours is the > first report saying that you followed the official guideline, > but you didn't state whether doing so actually helped (or whether > you just ported to 2.6 because the guideline told you to). Python 2.6 has other useful features, which I want to take advantage of, so getting py3k warnings is a bonus. Running 'python2.6 -3 setup.py test' over my code did in fact help clean up a couple of problems. Seems like a good first step when considering Python 3 support to me. >c) whatever 2.7 does (except perhaps for the warnings), it won't > be useful to applications, because they couldn't use it, anyway: > they need to support 2.4 and 2.5, and it won't have any of the > gimmicks people come up with for 2.7. Adding gimmicks to 2.7 > might actually hurt porting: people may have to special-case > 2.7 because their work-arounds for older versions may break in 2.7 > (e.g. testing whether a name is *not* defined, when it becomes > defined in 2.7), plus it gives them an incentive to not port > yet until they can drop support for anything before 2.7. > >Inherently, 2.8 can't improve on that. I'm not so sure. Yes, as a package maintainer there are older versions to think about, but time moves on for everyone (hopefully :). By the time 2.8 is released, what will be the minimum version of Python provided by most OS vendors (where the majority of Python users probably get their 'python')? I guess some people will have to support everything from Python 2.3 to 2.8 but you're talking supporting something like a spread of 7 years of Python versions. What other platform do you support for 7 years? Even Ubuntu long term support (LTS) releases only live for 5 years and even then, newer Pythons are often back ported. Or if not, then who cares? You're not going to use a newer version of a library on an LTS anyway. I'm not necessarily saying that justifies a 2.8 release. I'm just asking how we can make it easier for people to port to Python 3. The automatic 2to3 has already made it easier for me to port one library to Python 3 because I barely had to even think about it. Maybe that's enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 01:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:12:19 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <20100111191219.5fdd2542@freewill> On Jan 11, 2010, at 02:55 PM, Guido van Rossum wrote: >PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not >helpful. But I trust he has ported a lot more code to 3.x than I have >personally. Are there other experiences? I've only done one small library so far, but it was helpful to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From andrew at bemusement.org Tue Jan 12 01:07:26 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Tue, 12 Jan 2010 11:07:26 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100112000726.GO32351@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > I've done a fair bit of 3.x porting, and I'm firmly convinced that > 2.x can do nothing: [...] > Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. From solipsis at pitrou.net Tue Jan 12 01:28:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 00:28:26 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: Andrew Bennetts bemusement.org> writes: > > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. I don't think a 2.8 could make you much closer to 3.x. AFAICT, every possible way to ease transition to 3.x will already be in 2.7, or almost. But perhaps I'm lacking imagination. Regards Antoine. From brian.curtin at gmail.com Tue Jan 12 02:57:46 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 11 Jan 2010 19:57:46 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > -Brett > I think it would be interesting to see how people are using the tracker, or how they want to be using it. For example, there are currently over 1500 open issues with no stage set, some of which seemingly haven't been read by anyone at all. Would a properly set stage field save issues from falling into a black hole? Food for thought: according to the last tracker summary, there are over 1000 open issues with a patch, and issues stay open an average of 700 days (median 450). Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Jan 12 04:53:24 2010 From: jackdied at gmail.com (Jack Diederich) Date: Mon, 11 Jan 2010 22:53:24 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. [sidebar] 1) the metaclass fixer was a PITA to implement. 2) 95% of __metaclass__ definitions searchable via google code were of the "__metaclass__ = type" variety. The 2to3 patch exists only because of the few other uses. 3) 100% of the module level assignments in public projects were the "__metaclass__ = type" variety which is why there isn't a fixer for that. Also, a fixer would have been really, really ugly (munge every class definition in this module because there is a top level assignment). -Jack From benjamin at python.org Tue Jan 12 05:01:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 11 Jan 2010 22:01:40 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <1afaf6161001112001t5e7bab83t7d93f33fa11ce849@mail.gmail.com> 2010/1/11 Jack Diederich : > On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: >> As an example, the one library I've already ported used a metaclass. ?I don't >> see any way to specify that the metaclass should be used in a portable way. >> In Python 2.6 it's: >> >> class Foo: >> ? ?__metaclass__ = Meta >> >> and in Python 3 it's: >> >> class Foo(metaclass=Meta): >> >> 2to3 made that pain go away. > > [sidebar] > 1) the metaclass fixer was a PITA to implement. Does this make it any less useful, though? :) -- Regards, Benjamin From steven.bethard at gmail.com Tue Jan 12 06:57:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 11 Jan 2010 21:57:18 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 4:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. Actually there's a solution to this one too: FooBase = Meta('FooBase', (), {}) class Foo(FooBase): ... That should work in Python 2.X and 3.X. I've got argparse running on Python 2.3-3.1, and the changes were pretty easy. You can see them all in the revision here: http://code.google.com/p/argparse/source/detail?r=12 I have aspirations of putting all of the tricks I learned up up on the Wiki somewhere, but I just haven't had the time. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From regebro at gmail.com Tue Jan 12 07:30:10 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:30:10 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> On Mon, Jan 11, 2010 at 23:55, Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. But I trust he has ported a lot more code to 3.x than I have > personally. Are there other experiences? It doesn't warn for that many of the unportable problems, but I'm not sure it can warn for them either. It's certainly helpful, just not very much. :) I think the biggest help was added in 2.6.2, and that's warning for old integer division. It will also warn for modules that aren't supported anymore, if I remember correctly, and that's often helpful. But it won't warn for real tricky problems, like binary vs text in strings, and I don't see how it can either. And, I just realized, it doesn't warn for you using cmp or __cmp__ either, and 2to3 won't fix that, so it should actually warn for it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:49:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:49:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> On Tue, Jan 12, 2010 at 01:11, Barry Warsaw wrote: > Interesting. ?The only reason I never used it before is because I didn't know > about it. ?Am I alone? Maybe not, but the Distribute feature is there because IMO the distutils feature by itself isn't particularily useful. You need to write your own distutils extensions, in practice, and they are not trivial. Distribute has simply done it for you. :) > I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:53:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:53:20 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <319e029f1001112253x511f8109ja2300a022010ef99@mail.gmail.com> On Tue, Jan 12, 2010 at 01:07, Andrew Bennetts wrote: > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. ?I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. When 2.7 was discussed several people agreed that 2.7 should include as much 3.x stuff as possible to ease transition. That turned out to not be very much, so I'm not sure there is more. :) Unless of course, 2.8 starts including more of the refactored libraries, but that's a very minor issue in porting, so it won't help much. To really help, it needs to start implement things that break backwards compatibility. That would be weird. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Tue Jan 12 10:39:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:39:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4BA220.20701@voidspace.org.uk> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> <4B4BA220.20701@voidspace.org.uk> Message-ID: <4B4C435B.7080903@gmail.com> Michael Foord wrote: >> Note that the behaviour here is still different from that of a data >> descriptor: with a data descriptor, once it gets shadowed in the >> instance dictionary, the descriptor is ignored *completely*. The only >> way to get the descriptor involved again is to eliminate the shadowing. >> The non-data descriptor with only __set__ is just choosing not to >> override the attribute lookup process. >> >> > > Does that mean we need a third class of descriptors that are neither > data descriptors nor non-data descriptors? Not really - leaving out __get__ when defining __set__ just creates a data descriptor that happens to use the default attribute look-up process rather than defining a different one. (Note that I had the data/non-data terminology backwards in my previous message - I tend to think of the two kinds of descriptor as enforced and non-enforced respectively precisely because I have to think about it to remember that "functions are non-data descriptors and properties are data descriptors, hence non-data descriptors only define __get__ while data descriptors define __set__". I don't find the data/non-data terminology intuitive at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:44:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:44:18 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> Message-ID: <4B4C4472.10104@gmail.com> Lennart Regebro wrote: > And, I just realized, it doesn't warn for you using cmp or __cmp__ > either, and 2to3 won't fix that, so it should actually warn for it. I have a vague recollection that we tried to warn for that and ended up nixing the warning due to vast swarms of false alarms (or because you couldn't write correct 2.x code without triggering it). I'd be happy for someone to prove my recollection wrong though (i.e. by writing a patch that implements such warnings). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:48:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:48:57 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4C4589.70906@gmail.com> David Lyon wrote: >> This has nothing to do with pushing 3.x, but all with managing >> available manpower and still providing quality software. > > Python 3.x needs more carrots. As Guido has said a few times, the gains are far greater for *new* Python developers than they are for existing ones. Existing developers have to unlearn old habits and wait for libraries they already use to get ported. New developers can just get started with a much cleaner language. They don't have as rich a 3rd party ecosystem on Python 3 as they would on Python 2.x at this point in time, but unlike existing developers they also have the luxury of cherry-picking just those packages that already have Python 3 support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Jan 12 11:20:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:20:27 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit?: > > For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. I think most issues /have/ been read. It's just that for many of them, nobody is interested enough in or feels competent enough for fixing them. > Would a properly set stage field save issues from > falling into a black hole? I don't think this has anything to do with properly setting the stage field. We just have limited time and manpower. Perhaps one of our goals should be to reach out more to potential contributors. > Food for thought: according to the last tracker summary, there are over > 1000 open issues with a patch, and issues stay open an average of 700 > days (median 450). As for open issues with a patch, a "code review" button sending this patch to Rietveld (or any other similar tool) is something many of us have been hoping for :-) It would make reviewing easier, faster and more thorough (because you visualize things better than by looking at a diff). Regards Antoine. From solipsis at pitrou.net Tue Jan 12 11:36:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:36:49 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Tue, 12 Jan 2010 10:20:27 +0000, Antoine Pitrou a ?crit?: > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Speaking of which, Steve had something to say about it: http://holdenweb.blogspot.com/2009/11/comments-or-not-public-or- private.html cheers Antoine. From ncoghlan at gmail.com Tue Jan 12 13:10:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 22:10:14 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C66A6.2040601@gmail.com> Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : >> For example, there are currently over >> 1500 open issues with no stage set, some of which seemingly haven't been >> read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. There are actually a whole host of reasons issues can stagnate: - a feature request may seem reasonable (hence it doesn't get rejected outright), but the right API may not be clear (hence it doesn't get implemented in the near term) - a patch may be reviewed and found to have significant defects (or simply be overreaching the stated goal) but the original patch poster loses interest after meeting resistance in their ambition to fix something that is "obviously" broken - a problem may simply be hard to fix in a backwards compatible way (or even at all!) Ultimately it boils down to Antoine's two categories (lack of interest and lack of relevant expertise to make a final decision) but there are a lot of different ways to get into those two buckets. Because we aren't ruthless about pruning those kinds of issues out of the tracker they're the ones that are going to accumulate over time. I'd actually be interested to know what the issue stats are like when RFEs are excluded and when the search is limited to the items flagged as 'easy'. If easy bug fix issues are taking a long time to get closed than that would bother me a lot more than RFEs that sit on the tracker for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 12 13:14:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:14:47 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071447.675c8f24@freewill> On Jan 11, 2010, at 10:53 PM, Jack Diederich wrote: >3) 100% of the module level assignments in public projects were the >"__metaclass__ = type" variety which is why there isn't a fixer for >that. Also, a fixer would have been really, really ugly (munge every >class definition in this module because there is a top level >assignment). And almost certainly unnecessary. IME, those are all to easily make bare class definitions new style in Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 13:16:09 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:16:09 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071609.1dcfffa6@freewill> On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: >Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > >That should work in Python 2.X and 3.X. Ugly, but good call! :) >I've got argparse running on Python 2.3-3.1, and the changes were >pretty easy. You can see them all in the revision here: > > http://code.google.com/p/argparse/source/detail?r=12 > >I have aspirations of putting all of the tricks I learned up up on the >Wiki somewhere, but I just haven't had the time. The more resources we can provide people, both in code and in documentation, the better. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Tue Jan 12 13:29:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 12:29:12 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112071609.1dcfffa6@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> Message-ID: <4B4C6B18.7050008@voidspace.org.uk> On 12/01/2010 12:16, Barry Warsaw wrote: > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > >> Actually there's a solution to this one too: >> >> FooBase = Meta('FooBase', (), {}) >> class Foo(FooBase): >> ... >> >> That should work in Python 2.X and 3.X. >> > Ugly, but good call! :) > > There are all sorts of tricks. For example you can do exception handling that works with pre-2.6 syntax and 3.0 with a bare except and using sys.exc_info. It is horrible, but acceptable for short pieces of code (I have a couple of small modules that do this). I haven't yet tried converting larger code-bases to Python 3, but I think the workflow advocated by Martin is greatly preferable to the hacks and tricks needed to make the same codebase run under 2 & 3. Michael >> I've got argparse running on Python 2.3-3.1, and the changes were >> pretty easy. You can see them all in the revision here: >> >> http://code.google.com/p/argparse/source/detail?r=12 >> >> I have aspirations of putting all of the tricks I learned up up on the >> Wiki somewhere, but I just haven't had the time. >> > The more resources we can provide people, both in code and in documentation, > the better. > > Thanks! > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 12 14:09:50 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Jan 2010 14:09:50 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C749E.4040009@egenix.com> Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > * argparse (PEP 389) > * brief mention on still wanting to break out the stdlib from CPython > * an official policy on extension modules? (i.e. must have a pure Python > implementation which can mean a ctypes implementation unless given an > explicit waiver) > > That's everything from a stdlib perspective. I have half-baked ideas, but > nothing concrete enough to bring up unless people really want to discuss > stuff like how to potentially standardize module deprecation warnings, etc. > > In terms of me personally, I do plan to bring up at some point during the > summit these points which don't squarely fit during my time slot: > > * an official unofficial policy on how new proposed features should come to > us (i.e. working code to python-ideas with a shell of a PEP that includes > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > If there is something missing from the stdlib discussion that you think > should be brought up at the summit let me know. And if there is something > here you want to discuss before the summit let me know and I can start a > separate thread on it. Could you please put these things and the results up on the Python wiki ?! We're going to have a language summit at EuroPython this year as well and may want to continue/extend the discussion based on what you're doing at PyCon. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 12 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brian.curtin at gmail.com Tue Jan 12 15:33:06 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 12 Jan 2010 08:33:06 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Tue, Jan 12, 2010 at 04:20, Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : > > > > For example, there are currently over > > 1500 open issues with no stage set, some of which seemingly haven't been > > read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. > > > Would a properly set stage field save issues from > > falling into a black hole? > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Agreed, I didn't mean to place blame on the stage field, I just ran with how I view that field since it was mentioned. When I'm thinking in "code-writing developer mode" (tm), I'm more likely to go to issues which have a stage so I know what I'm going into -- what needs to be worked on. When I'm in project cleanup mode, I go by stageless issues -- what is necessary for this to begin or end work. Maybe others work differently...software projects take all kinds :-) Anyways, sorry for the off-topic. If this is a summit worthy discussion, cool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 17:12:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 11:12:28 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C66A6.2040601@gmail.com> References: <4B4C66A6.2040601@gmail.com> Message-ID: <20100112161228.300EA1D688D@kimball.webabinitio.net> On Tue, 12 Jan 2010 22:10:14 +1000, Nick Coghlan wrote: > There are actually a whole host of reasons issues can stagnate: > - a feature request may seem reasonable (hence it doesn't get rejected > outright), but the right API may not be clear (hence it doesn't get > implemented in the near term) > - a patch may be reviewed and found to have significant defects (or > simply be overreaching the stated goal) but the original patch poster > loses interest after meeting resistance in their ambition to fix > something that is "obviously" broken > - a problem may simply be hard to fix in a backwards compatible way (or > even at all!) I would add: - a patch is reviewed but the reviewer requests a second opinion before committing, which never arrives, and the original reviewer forgets about the issue. I have occasionally observed apparently moribund issues spring to life and get resolved when a triage person does something as simple as updating the releases to which an issue applies. > Ultimately it boils down to Antoine's two categories (lack of interest > and lack of relevant expertise to make a final decision) but there are a > lot of different ways to get into those two buckets. There's a third bucket here: lack of time. Sometimes there is interest and expertise, but no time. Worse, sometimes we end up waiting on the person with the expertise and interest but no time, when someone else with somewhat less expertise but more time should just go ahead and do the commit. (*That* one should be fixable.) > Because we aren't ruthless about pruning those kinds of issues out of > the tracker they're the ones that are going to accumulate over time. Another other category that hangs around in the tracker is items for which there simply isn't a consensus. I suppose those could be brought to python-dev for a final decision if someone wanted to tackle that task. And I don't think it is a lack of ruthlessness. I think it is the current community consensus that we leave these issues open in the tracker in case someone does come along and want to deal with them. (*) > I'd actually be interested to know what the issue stats are like when > RFEs are excluded and when the search is limited to the items flagged as > 'easy'. If easy bug fix issues are taking a long time to get closed than > that would bother me a lot more than RFEs that sit on the tracker for > years. I'd also be interested to know what the average and median lifetime of closed bug reports is. Not the metrics for open issues, but the metrics for closed ones. My theory is that we close a lot of bugs fairly promptly, but the ones in the above categories make the average age of *open* issues high. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls (*) For example, I'm currently going through all the open issues relating to the email package, and some of them are pretty darn old, yet still have useful content. From brett at python.org Tue Jan 12 18:40:13 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:40:13 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C749E.4040009@egenix.com> References: <4B4C749E.4040009@egenix.com> Message-ID: On Tue, Jan 12, 2010 at 05:09, M.-A. Lemburg wrote: > Brett Cannon wrote: > > Michael has given me the hg transition/stdlib time slot at the language > > summit this year. In regards to that I plan to lead a discussion on: > > > > * where we are at w/ the Hg transition (Dirkjan should be there and I did > a > > blog post on this topic recently: > > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > > * argparse (PEP 389) > > * brief mention on still wanting to break out the stdlib from CPython > > * an official policy on extension modules? (i.e. must have a pure Python > > implementation which can mean a ctypes implementation unless given an > > explicit waiver) > > > > That's everything from a stdlib perspective. I have half-baked ideas, but > > nothing concrete enough to bring up unless people really want to discuss > > stuff like how to potentially standardize module deprecation warnings, > etc. > > > > In terms of me personally, I do plan to bring up at some point during the > > summit these points which don't squarely fit during my time slot: > > > > * an official unofficial policy on how new proposed features should come > to > > us (i.e. working code to python-ideas with a shell of a PEP that includes > > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > > * any changes needed to the issue tracker to help with the workflow? > (stage > > field seems like a failed experiment and we now have several effective > > triage people who can help w/ guiding changes) > > > > If there is something missing from the stdlib discussion that you think > > should be brought up at the summit let me know. And if there is something > > here you want to discuss before the summit let me know and I can start a > > separate thread on it. > > Could you please put these things and the results up on the Python > wiki ?! > > We're going to have a language summit at EuroPython this year > as well and may want to continue/extend the discussion based > on what you're doing at PyCon. > > I expect there will be at least summary emails on what gets discussed. There is also a chance that it will be videotaped. -Brett > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 12 2010) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 18:47:50 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:47:50 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? >> (stage field seems like a failed experiment and we now have several >> effective triage people who can help w/ guiding changes) >> >> -Brett >> > I think it would be interesting to see how people are using the tracker, or > how they want to be using it. For example, there are currently over 1500 > open issues with no stage set, some of which seemingly haven't been read by > anyone at all. Would a properly set stage field save issues from falling > into a black hole? > > "Using" is the key word there. I know this thread went on about how bugs tend to end up being left open, but what I was proposing to talk about was whether there is any shift desired by the seasoned tracker users to help in their work. For instance, the Stage field was meant to help, but I don't think it is really getting used much, which makes it at best just a UI junk and at worst something to confuse new users. So I just wanted to discuss things as a group to see if we could streamline some things, add others, etc. At worst I will try to grab people like Mark D., R. David, Antoine, Georg, etc. at the summit (not sure exactly which the heavy bug closers will be there), take them aside, and flat-out ask what they need to make their jobs easier. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 19:29:06 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:29:06 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C6B18.7050008@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> <4B4C6B18.7050008@voidspace.org.uk> Message-ID: On Tue, Jan 12, 2010 at 04:29, Michael Foord wrote: > On 12/01/2010 12:16, Barry Warsaw wrote: > > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > > > Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > > That should work in Python 2.X and 3.X. > > > Ugly, but good call! :) > > > > > There are all sorts of tricks. For example you can do exception handling > that works with pre-2.6 syntax and 3.0 with a bare except and using > sys.exc_info. It is horrible, but acceptable for short pieces of code (I > have a couple of small modules that do this). > > I haven't yet tried converting larger code-bases to Python 3, but I think > the workflow advocated by Martin is greatly preferable to the hacks and > tricks needed to make the same codebase run under 2 & 3. > > In other words we need to pull together a HOWTO for Python source like the one for extension modules that Benjamin wrote and make it rather prominently linked from the Python 3 documentation index page. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 19:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 13:31:23 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? > >> (stage field seems like a failed experiment and we now have several > >> effective triage people who can help w/ guiding changes) > >> > > I think it would be interesting to see how people are using the tracker, or > > how they want to be using it. For example, there are currently over 1500 > > open issues with no stage set, some of which seemingly haven't been read by > > anyone at all. Would a properly set stage field save issues from falling > > into a black hole? > > > "Using" is the key word there. I know this thread went on about how bugs > tend to end up being left open, but what I was proposing to talk about was > whether there is any shift desired by the seasoned tracker users to help in > their work. For instance, the Stage field was meant to help, but I don't > think it is really getting used much, which makes it at best just a UI junk > and at worst something to confuse new users. So I just wanted to discuss > things as a group to see if we could streamline some things, add others, > etc. Well, I for one find the stage field useful. It reminds me at a glance where the issue is at in the workflow (and 'not set' is a valid place in the workflow that an issue sometimes stays in for a while for reason other than not having been triaged yet). I suspect that if we discussed it as a group (we who make the most changes on the tracker) we could improve it, and the interface in general. By the way, you could talk to people who aren't going to be at the summit on #python-dev; I think all the currently tracker-active people hang out there on a regular basis. I'll have to give some thought to what changes/improvements might be most useful, now that I've been doing this for a while. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Tue Jan 12 19:34:02 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:34:02 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> References: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> Message-ID: On Tue, Jan 12, 2010 at 10:31, R. David Murray wrote: > On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > > On Mon, Jan 11, 2010 at 17:57, Brian Curtin > wrote: > > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > > >> * any changes needed to the issue tracker to help with the workflow? > > >> (stage field seems like a failed experiment and we now have several > > >> effective triage people who can help w/ guiding changes) > > >> > > > I think it would be interesting to see how people are using the > tracker, or > > > how they want to be using it. For example, there are currently over > 1500 > > > open issues with no stage set, some of which seemingly haven't been > read by > > > anyone at all. Would a properly set stage field save issues from > falling > > > into a black hole? > > > > > "Using" is the key word there. I know this thread went on about how bugs > > tend to end up being left open, but what I was proposing to talk about > was > > whether there is any shift desired by the seasoned tracker users to help > in > > their work. For instance, the Stage field was meant to help, but I don't > > think it is really getting used much, which makes it at best just a UI > junk > > and at worst something to confuse new users. So I just wanted to discuss > > things as a group to see if we could streamline some things, add others, > > etc. > > Well, I for one find the stage field useful. It reminds me at a glance > where the issue is at in the workflow (and 'not set' is a valid place > in the workflow that an issue sometimes stays in for a while for reason > other than not having been triaged yet). I suspect that if we discussed > it as a group (we who make the most changes on the tracker) we could > improve it, and the interface in general. > > By the way, you could talk to people who aren't going to be at the > summit on #python-dev; I think all the currently tracker-active people > hang out there on a regular basis. > > Sounds reasonable. > I'll have to give some thought to what changes/improvements might be > most useful, now that I've been doing this for a while. How about everyone who is active on bugs.python.org give it some thought and we can try to have a discussion at some point in the relatively near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jan 12 21:58:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 06:58:49 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: <4B4C749E.4040009@egenix.com> Message-ID: <4B4CE289.6040709@gmail.com> Brett Cannon wrote: > I expect there will be at least summary emails on what gets discussed. > There is also a chance that it will be videotaped. The Wiki makes for a better summary archive though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jan 12 22:53:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:53:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <4B4CEF3D.8000107@v.loewis.de> >> a) telling people that they have to move to 2.6 first actually >> hurts migration, instead of helping, because it implies to them >> that they have to drop old versions (e.g. 2.3.) - just because >> they had *always* dropped old versions before supporting new ones. > > Is it just an implication, or is it reality? That's only the implication. However, this was precisely the dialogue when talking to Django. If you start with "start supporting 2.6", the immediate response, without listening further, was, "ok, wait until we drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). Then explain it to the individual you are talking to, wait for the next developer of the project step along, and see how he brings up the very same line of thinking (supporting new versions == dropping support for old versions). I think only part of that comes from the maintenance burden. The other part is that they *want* to drop support for old versions, so that they can eventually start using new features (e.g. generator expressions). So they welcome the requirement to support new versions as an excuse to drop old ones ("it is obvious that you have to drop 2.3 to support 3.2"). However, their users then won't let them drop old versions. > >> b) IMO, people also don't gain much by first migrating to 2.6. >> In principle, it gives them the opportunity to get py3k warnings. >> However, I haven't heard a single positive report where these >> warnings have actually helped people in porting. Yours is the >> first report saying that you followed the official guideline, >> but you didn't state whether doing so actually helped (or whether >> you just ported to 2.6 because the guideline told you to). > > Python 2.6 has other useful features, which I want to take advantage of I think you are a minority with that, being able to actually use the 2.6 features already. Many projects can't, as they have to support at least 2.4 still (so the with statement is right out). >> Inherently, 2.8 can't improve on that. > > I'm not so sure. Yes, as a package maintainer there are older versions to > think about, but time moves on for everyone (hopefully :). By the time 2.8 is > released, what will be the minimum version of Python provided by most OS > vendors (where the majority of Python users probably get their 'python')? "Current" Linux distributions will have 2.6 then. "Old" installations will have 2.4. > I > guess some people will have to support everything from Python 2.3 to 2.8 but > you're talking supporting something like a spread of 7 years of Python > versions. What other platform do you support for 7 years? I think 2.3 will really be gone by the time 2.8 might get released. Even with 2.7, you'd end up with a span of seven years, though. Python had been supporting Windows 95 for more than 7 years (I think rather 9 or 10), likewise Windows 3.1 before that. Python 2.7 will likely still support Windows 2000, which then will be ten years old. Solaris support will probably go back to Solaris 2.6, which will be 13 years old when Python 2.7 gets released. It's only the Linux (and OS X) releases that move so quickly. Regards, Martin From martin at v.loewis.de Tue Jan 12 22:56:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:56:55 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> Message-ID: <4B4CF027.4080007@v.loewis.de> > Maybe not, but the Distribute feature is there because IMO the > distutils feature by itself isn't particularily useful. You need to > write your own distutils extensions, in practice, and they are not > trivial. I wouldn't say that. My Django port works with bare distutils (as does Django itself), and it works fine. That distribute had to redo it is only because setuptools *replaces* the build_py command, as does the 2to3 support in distutils. So only if you have a different build_py already, you can't use what is in distutils. Regards, Martin From fuzzyman at voidspace.org.uk Tue Jan 12 23:02:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:02:34 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CEF3D.8000107@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> Message-ID: <4B4CF17A.1000503@voidspace.org.uk> On 12/01/2010 21:53, "Martin v. L?wis" wrote: >>> a) telling people that they have to move to 2.6 first actually >>> hurts migration, instead of helping, because it implies to them >>> that they have to drop old versions (e.g. 2.3.) - just because >>> they had *always* dropped old versions before supporting new ones. >>> >> Is it just an implication, or is it reality? >> > That's only the implication. However, this was precisely the dialogue > when talking to Django. If you start with "start supporting 2.6", the > immediate response, without listening further, was, "ok, wait until we > drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). > > Then explain it to the individual you are talking to, wait for the next > developer of the project step along, and see how he brings up the > very same line of thinking (supporting new versions == dropping support > for old versions). > > I think only part of that comes from the maintenance burden. The other > part is that they *want* to drop support for old versions, so that they > can eventually start using new features (e.g. generator expressions). > So they welcome the requirement to support new versions as an excuse > to drop old ones ("it is obvious that you have to drop 2.3 to support > 3.2"). However, their users then won't let them drop old versions. > > > I agree with Martin that the *perception* is that to use Python 2.6 to help you port to Python 3 you have to be willing to drop support for earlier versions of Python. >> >>> b) IMO, people also don't gain much by first migrating to 2.6. >>> In principle, it gives them the opportunity to get py3k warnings. >>> However, I haven't heard a single positive report where these >>> warnings have actually helped people in porting. Yours is the >>> first report saying that you followed the official guideline, >>> but you didn't state whether doing so actually helped (or whether >>> you just ported to 2.6 because the guideline told you to). >>> >> Python 2.6 has other useful features, which I want to take advantage of >> > I think you are a minority with that, being able to actually use the 2.6 > features already. Many projects can't, as they have to support at least > 2.4 still (so the with statement is right out). > > Well, the IronPython community has almost completely moved over to IronPython 2.6. :-) We tend to ship our Python runtime with our applications though. As it happens I'm now working with CPython on the server (Python 2.5) and IronPython in the browser where we are using 2.6. The new property feature is nice, as is having with without a __future__ import. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From regebro at gmail.com Tue Jan 12 23:03:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 23:03:41 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF027.4080007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> <4B4CF027.4080007@v.loewis.de> Message-ID: <319e029f1001121403x14ef7ec8x729fb80fa67feaef@mail.gmail.com> On Tue, Jan 12, 2010 at 22:56, "Martin v. L?wis" wrote: >> Maybe not, but the Distribute feature is there because IMO the >> distutils feature by itself isn't particularily useful. You need to >> write your own distutils extensions, in practice, and they are not >> trivial. > > I wouldn't say that. My Django port works with bare distutils (as > does Django itself), and it works fine. > > That distribute had to redo it is only because setuptools *replaces* > the build_py command, as does the 2to3 support in distutils. So only > if you have a different build_py already, you can't use what is in > distutils. Yeah, you are right, I misremembered. The actual additional feature is the support for the test command. Testing under Python 2 and 3 without it is annoying. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 12 23:04:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:04:37 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <4B4CF1F5.4050403@v.loewis.de> > [...] >> I've done a fair bit of 3.x porting, and I'm firmly convinced that >> 2.x can do nothing: > [...] >> Inherently, 2.8 can't improve on that. > > I agree that there are limitations like the ones you've listed, but I > disagree with your conclusion. Maybe you assume that it's just as hard > to move to 2.8 (using the py3k backports not available in say 2.5) as it > is to 3.x? Not at all, no. I'd rather expect that code that runs on 2.7 will run on 2.8 unmodified. > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? > I think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? > Then, when all of your dependencies (or viable alternatives to those > dependencies) are available for 3.x, you'll have an easier transition if > you can start from a 2.x with fewer differences in features. But you won't *have* fewer differences. Just because your code runs on 2.8 doesn't mean it will stop running on 2.3 (if you have a need for that). This doesn't get you any closer - you can't use any of the 2.8 features as long as you have to support older versions of Python. > Fundamentally the more 2.x can converge on 3.x, the easier it will be > for users to make the leap, because it will be a smaller leap. No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. > The > longer the 2.x series lives, the more these newer 2.x versions like 2.7 > and maybe even 2.8 will be available on common platforms for people to > depend upon as minimum versions, which means that as time goes by they > can depend on a version that's closer to 3.x. No, that's incorrect. Suppose 2.7 is the last 2.x release, to be released in 2010. Then, in 2015, it may be that everybody has migrated to 2.7, which then is a common platform. If you release 2.8 in 2012, then, in 2015, people will be split between 2.7 and 2.8, and so there won't be a common platform before 2017. So stopping 2.x development *earlier* will also give us a common platform earlier. Regareds, Martin From martin at v.loewis.de Tue Jan 12 23:07:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:07:02 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4CF286.5040002@v.loewis.de> > I think it would be interesting to see how people are using the tracker, > or how they want to be using it. For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. Would a properly set stage field save issues from > falling into a black hole? I personally don't think this would be the case. What would help is people who actually *work* on the issues, rather than merely reading them. However, this being open source, there is no way to force it (unless you create an incentive, such as the five-for-one deal). Regards, Martin From python at mrabarnett.plus.com Tue Jan 12 23:10:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jan 2010 22:10:28 +0000 Subject: [Python-Dev] regex module Message-ID: <4B4CF354.2050603@mrabarnett.plus.com> Hi all, I'm back on the regex module after doing other things and I'd like your opinion on a number of matters: Firstly, the current re module has a bug whereby it doesn't split on zero-width matches. The BDFL has said that this behaviour should be retained by default in case any existing software depends on it. My question is: should my regex module still do this for Python 3? Speaking personally, I'd like it to behave correctly, and Python 3 is the version where backwards-compatibility is allowed to be broken. Secondly, Python 2 is reaching the end of the line and Python 3 is the future. Should I still release a version that works with Python 2? I'm thinking that it could be confusing if new regex module did zero-width splits correctly in Python 3 but not in Python 2. And also, should I release it only for Python 3 as a 'carrot'? Finally, the module allows some extra backslash escapes, eg \g, in the pattern. Should it treat ill-formed escapes, eg \g, as it would have treated them in the re module? Thanks From michael at voidspace.org.uk Tue Jan 12 23:46:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:46:49 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 Message-ID: <4B4CFBD9.1090009@voidspace.org.uk> I presume the email below is about the Windows binary. Does the AMD64 release work on intel 64bit and can we make the wording clearer on the download page? The current description is " Windows AMD64 binary". All the best, Michael -------- Original Message -------- Subject: Download Page - AMD64 Date: Fri, 11 Dec 2009 04:03:16 -0600 From: Thomas Brownback To: webmaster at python.org Should I use the AMD64 version of Python on an Intel 64 chip? I know those 64-bit implementations are very similar, but are they similar enough that your AMD64 will work on Intel? If so, perhaps a note should be placed on that page to help avoid confusion. Explicit being better than implicit and all. Thanks for your time, Thomas -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at bemusement.org Tue Jan 12 23:49:56 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Wed, 13 Jan 2010 09:49:56 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100112224956.GC28576@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > But a hypothetical 2.8 would also give people a way to move closer to > > py3k without giving up on using all their 2.x-only dependencies. > > How so? If they use anything that is new in 2.8, they *will* need to > drop support for anything before it, no??? > > > I think it's much more likely that libraries like Twisted can support 2.8 > > in the near future than 3.x. > > Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does > that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. From tjreedy at udel.edu Tue Jan 12 23:51:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 17:51:31 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: On 1/12/2010 5:04 PM, "Martin v. L?wis" wrote: > But you won't *have* fewer differences. Just because your code runs > on 2.8 doesn't mean it will stop running on 2.3 (if you have a need > for that). This doesn't get you any closer - you can't use any of > the 2.8 features as long as you have to support older versions of > Python. > >> Fundamentally the more 2.x can converge on 3.x, the easier it will be >> for users to make the leap, because it will be a smaller leap. > > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. > >> The >> longer the 2.x series lives, the more these newer 2.x versions like 2.7 >> and maybe even 2.8 will be available on common platforms for people to >> depend upon as minimum versions, which means that as time goes by they >> can depend on a version that's closer to 3.x. > > No, that's incorrect. Suppose 2.7 is the last 2.x release, to be > released in 2010. Then, in 2015, it may be that everybody has migrated > to 2.7, which then is a common platform. > > If you release 2.8 in 2012, then, in 2015, people will be split between > 2.7 and 2.8, and so there won't be a common platform before 2017. Just like people today may need to work with both 2.5 and 2.6, or privately backport 2.6 bugfixes to 2.5. > So stopping 2.x development *earlier* will also give us a common > platform earlier. With years of bug fixes and hence high quality. From tjreedy at udel.edu Wed Jan 13 00:05:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 18:05:12 -0500 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On 1/12/2010 5:10 PM, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. Are you writing a new module with a new name? If so, do you expect it to replace or augment re? (This is the same question as for optparse vs. argparse, which I understand to not yet be decided.) > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, 2.7 stdlib is already out. A new engine should get some community testing before going in the stdlib. Even 3.2 beta is not that far off (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? What does re do with analogous cases? Terry Jan Reedy From david.lyon at preisshare.net Wed Jan 13 00:20:54 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 13 Jan 2010 10:20:54 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C4589.70906@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> <4B4C4589.70906@gmail.com> Message-ID: <1333.218.214.45.58.1263338454.squirrel@syd-srv02.ezyreg.com> > Nick wrote: >>> This has nothing to do with pushing 3.x, but all with managing >>> available manpower and still providing quality software. >> >> Python 3.x needs more carrots. > > As Guido has said a few times, the gains are far greater for *new* > Python developers than they are for existing ones. Well both you and Guido are most likely 100% correct. :-) > They don't have as rich a 3rd party ecosystem > on Python 3 as they would on Python 2.x at this point in time, but > unlike existing developers they also have the luxury of cherry-picking > just those packages that already have Python 3 support. Most likely. I wouldn't want to say anything to discourage people from going to python 3. In other languages, I have much experience of making the jump to 'a whole new world'. It's unsettling at first, but after that, you suddenly find the new model has a better turbo than the last and you're at the next corner faster than you can think. So it's all good. But I still maintain Python 3.0 needs more carrots. For example, if mercurial or any other cool lib gets added to python 3 (and I can name a few that should go in python 3) then they should be added to python 3 and not to python 2.x They would serve as good carrots. Make fresh the python 3 stdlib and preserve the python 2.x stdlib. I really think we are somewhat short on resources to do what Guido has asked about bringing python up to CPAN level with respect to packages. We're starting a long way back with horses and swords and trying to catch a well fed and greased modern machine.. I'm sure we can get a modern package testbot operational for python. But I wish those who were complaining about the packaging problem so much could throw some money at the problem instead of moaning about it. As is done on other open-source projects. Some organisation would be beneficial. Finding funds so that a small group of people could work on the problem would be a great boost to forward progress. I just think python packaging needs six months of that to repair the years and years of neglect and stagnation. Even if it is only beer and bus fare money. It just needs a temporary shot of adrenalin in the arm.. so to speak.. Regards David From martin at v.loewis.de Wed Jan 13 00:28:14 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 00:28:14 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D058E.4030404@v.loewis.de> > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? "intel 64bit" is as clear as mud. It could mean the "Intel 64" architecture, or it could mean the "IA-64" architecture, both are 64-bit architectures with processors manufactured by Intel. The former is officially called AMD64 (not by Intel, but officially), and our AMD64 binaries run on such processors. The latter is also called "Itanium", and we stopped producing binaries for that architecture a while ago; the AMD64 binaries will *not* run on it. The wording could probably be changed to match the reader's expectation more, most likely, it would also get more incorrect in the process. To make it correct and explicit, an entire paragraph educating users about 64-bit architectures and that there are many of them and that they are mutually incompatible might be required. Most likely, Thomas' processor implements the AMD64 architecture, even though it is manufactured by Intel. A short sentence that he would probably understand (given that he used "Intel 64", not "64bit") is: """The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).""" Regards, Martin From martin at v.loewis.de Wed Jan 13 00:30:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 00:30:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112224956.GC28576@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> <20100112224956.GC28576@steerpike.home.puzzling.org> Message-ID: <4B4D0613.1010503@v.loewis.de> > I'm not talking about Twisted moving to 3.x (FWIW, I think the only > movement there so far is some patches for some -3 warnings). The > situation I'm describing is a project X that: > > (a) has 2.x-only dependencies, and > (b) would like to be as close as possible to 3.x (because they like > the new features and/or want to be as ready as possible to jump > when (a) is fixed). This assumes that jumping to 3.x is easier if you are closer to it. Please trust me that this is not the case. >> No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they >> likely won't. > > But this is my point. I think they would as an intermediate step to > jumping to 3.x (which also requires dropping 2.5, after all!), if for > some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. No, and no. No: it's not an intermediate step, and no, supporting 3.x does *not* require dropping 2.5. Regards, Martin From fuzzyman at voidspace.org.uk Wed Jan 13 00:40:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:40:35 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D058E.4030404@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> Message-ID: <4B4D0873.5070006@voidspace.org.uk> On 12/01/2010 23:28, "Martin v. L?wis" wrote: > [snip...] > """The binaries for AMD64 will also work on processors that implement > the Intel 64 architecture (formerly EM64T), i.e. the architecture that > Microsoft calls x64, and AMD called x86-64 before calling it AMD64. > They will not work on Intel Itanium Processors (formerly IA-64).""" > > To those not intimately aware of the history and present of 64 bit architecture this sentence would be a great addition - thanks. I'm adding it now as a footnote. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From lists at cheimes.de Wed Jan 13 00:41:04 2010 From: lists at cheimes.de (Christian Heimes) Date: Wed, 13 Jan 2010 00:41:04 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D0890.2030505@cheimes.de> Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". The installer works on all AMD64 compatible Intel CPUs. *scnr* As you most likely know all modern Intel 64bit CPUs are based on AMD's x86-64 design. Only the Itanium family is based on the other Intel 64bit design: IA-64. The name AMD64 was chosen because most Linux and BSD distributions call it so. The name 'AMD64' has caused confusion in the past because users thought that the installer works only on AMD CPUs. How about: * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / X86-64 binary -- does not include source) instead of: * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does not include source) ? Christia From fuzzyman at voidspace.org.uk Wed Jan 13 00:55:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:55:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0873.5070006@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> <4B4D0873.5070006@voidspace.org.uk> Message-ID: <4B4D0BD4.5050109@voidspace.org.uk> On 12/01/2010 23:40, Michael Foord wrote: > On 12/01/2010 23:28, "Martin v. L?wis" wrote: >> [snip...] >> """The binaries for AMD64 will also work on processors that implement >> the Intel 64 architecture (formerly EM64T), i.e. the architecture that >> Microsoft calls x64, and AMD called x86-64 before calling it AMD64. >> They will not work on Intel Itanium Processors (formerly IA-64).""" >> > > To those not intimately aware of the history and present of 64 bit > architecture this sentence would be a great addition - thanks. I'm > adding it now as a footnote. > I can add it now to the main download page and existing release pages - but are there changes needed to any scripts to change pages created for *new* releases? All the best, Michael Foord > All the best, > > Michael Foord > >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From python at mrabarnett.plus.com Wed Jan 13 01:22:01 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 00:22:01 +0000 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D1229.70708@mrabarnett.plus.com> Terry Reedy wrote: > On 1/12/2010 5:10 PM, MRAB wrote: >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. > > Are you writing a new module with a new name? If so, do you expect it to > replace or augment re? (This is the same question as for optparse vs. > argparse, which I understand to not yet be decided.) > It's a module called 'regex'. It can be used in place of 're' by using "import regex as re", except for differences such as "\g" being a legal group reference in pattern strings. >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? > > 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, > 2.7 stdlib is already out. A new engine should get some community > testing before going in the stdlib. Even 3.2 beta is not that far off > (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? > > What does re do with analogous cases? > The 're' module treats r"\g" as "g"; both 're' and 'regex' treat, say, r"\q" as "q". The closest analogue to what I'm asking about is that re treats the ill-formed repeat r"x{1," as a literal, which sort of suggests that r"\g" should be treated as "g", but r"\g" is now a group reference (re would treat that as "g". Does that sound reasonable? From fuzzyman at voidspace.org.uk Wed Jan 13 01:50:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 00:50:39 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D18DF.6070502@voidspace.org.uk> On 12/01/2010 23:41, Christian Heimes wrote: > Michael Foord wrote: > >> I presume the email below is about the Windows binary. Does the AMD64 >> release work on intel 64bit and can we make the wording clearer on the >> download page? >> >> The current description is " Windows AMD64 binary". >> > The installer works on all AMD64 compatible Intel CPUs. *scnr* > > As you most likely know all modern Intel 64bit CPUs are based on AMD's > x86-64 design. Only the Itanium family is based on the other Intel 64bit > design: IA-64. The name AMD64 was chosen because most Linux and BSD > distributions call it so. The name 'AMD64' has caused confusion in the > past because users thought that the installer works only on AMD CPUs. > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) > Right - I've made that change for the Python 2.6, 2.7, 3.0 and 3.1 download pages with a footnote. Prior to that we were offering ia64 *and* amd64 releases so I've left those unchanged. Thanks, Michael > ? > > Christia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From exarkun at twistedmatrix.com Wed Jan 13 02:40:03 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 13 Jan 2010 01:40:03 -0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100113014003.1898.1305834328.divmod.xquotient.219@localhost.localdomain> On 12 Jan, 10:04 pm, martin at v.loewis.de wrote: >>[...] >>>I've done a fair bit of 3.x porting, and I'm firmly convinced that >>>2.x can do nothing: >>[...] >>>Inherently, 2.8 can't improve on that. >> >>I agree that there are limitations like the ones you've listed, but I >>disagree with your conclusion. Maybe you assume that it's just as >>hard >>to move to 2.8 (using the py3k backports not available in say 2.5) as >>it >>is to 3.x? > >Not at all, no. I'd rather expect that code that runs on 2.7 will run >on 2.8 unmodified. >>But a hypothetical 2.8 would also give people a way to move closer to >>py3k without giving up on using all their 2.x-only dependencies. > >How so? If they use anything that is new in 2.8, they *will* need to >drop support for anything before it, no??? >>I think it's much more likely that libraries like Twisted can support >>2.8 >>in the near future than 3.x. > >Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does >that help Twisted in moving to 3.2? I'm not reading this thread carefully enough to make any arguments on either side, but I can contribute a fact. Twisted very likely does not support 2.8 today. I base this on the fact that Twisted does not support 2.7 today, and I expect 2.8 will be more like 2.7 than it will be like 2.3 - 2.6 (which Twisted does support). When I say "support" here, I mean "all of the Twisted unit tests pass on it". Jean-Paul From tseaver at palladion.com Wed Jan 13 03:57:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jan 2010 21:57:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4D36AA.7020607@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karen Tracey wrote: > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> >> > Well yay. Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference. That's > down from 104 failures and 40 errors with 2.7 alpha 1. > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Just to add another "success" data point: Zope2's trunk, as well as the 2.12 release, passes all tests (2535 on the trunk) and brings up the appserver just fine under 2.7a2. There is an obnoxious deprecation warning out of the distutils: DeprecationWarning: 'compiler' specifies the compiler type in build_ext. If you want to get the compiler object itself, use 'compiler_obj' which is likely a simple one-line fix, if I only knew what the hell it was whining about. ;) The warning is extra obnoxious because it doesn't tell me what in *my* code triggers the warning (it (needs a 'stacklevel' argument). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktNNqQACgkQ+gerLs4ltQ7d5gCcCGKVjyOlxnrAln0UnRibS7kd lNIAoIs1RlSGMtJWaY11BqptfDmQvR87 =mIOO -----END PGP SIGNATURE----- From python at mrabarnett.plus.com Wed Jan 13 04:09:34 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 03:09:34 +0000 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D396E.4050500@mrabarnett.plus.com> MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > I've just noticed something odd about the re module: the sub() method doesn't take 'pos' or 'endpos' arguments. search() does; match() does; findall() does(); finditer() does; but sub() doesn't. Maybe there has never been a demand for it. (Nor split(), for that matter.) From brett at python.org Wed Jan 13 04:58:11 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 19:58:11 -0800 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On Tue, Jan 12, 2010 at 14:10, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > If it is a separate module under a different name it can do the proper thing. People will just need to be aware of the difference when they import the module. > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > That's totally up to you. There is practically no chance of it getting into the 2.x under the stdlib at this point since 2.7b1 is coming up and this module has not been out in the wild for a year (to my knowledge). If you want to support 2.x that's fine and I am sure users would appreciate it, but it isn't necessary to get into the Python 3 stdlib. > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > > If you want to minimize the differences then it should probably match. As I said, since it is a different name to import under it can deviate where reasonable, just make sure to clearly document the deviations. -Brett > Thanks > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jan 13 07:11:49 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 07:11:49 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D6425.3060307@v.loewis.de> > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) -1. AMD doesn't want us to use the term x86-64 anymore, but wants us to use AMD64 instead. I think we should comply - they invented the architecture, so they have the right to give it a name. Neither Microsoft nor Intel have such a right. Regards, Martin From sridharr at activestate.com Wed Jan 13 07:47:38 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 12 Jan 2010 22:47:38 -0800 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D6C8A.7070307@activestate.com> On 1/12/2010 2:46 PM, Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". FWIW, we simply use (64-bit, x64). Platform Download ============================================== Windows (x86) Windows Installer (MSI) Windows (64-bit, x64) Windows Installer (MSI) https://www.activestate.com/activepython/downloads/ -srid From solipsis at pitrou.net Wed Jan 13 08:08:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Jan 2010 07:08:55 +0000 (UTC) Subject: [Python-Dev] Fwd: Download Page - AMD64 References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) +1. I don't care about trademarks or official names, we should call it whatever is obvious for our users. As for Itanium, it is practically dead, I think there is little chance of people mixing it up with x86-64. cheers Antoine. From michael at voidspace.org.uk Wed Jan 13 10:32:00 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 09:32:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4D9310.6060907@voidspace.org.uk> On 13/01/2010 06:11, "Martin v. L?wis" wrote: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) >> >> instead of: >> >> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >> not include source) >> > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. > I think we should use whatever is most informative and least confusing to our users - we owe our allegiance to them and not to a processor vendor. Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Wed Jan 13 11:30:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:30:50 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF17A.1000503@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> Message-ID: <4B4DA0DA.7070007@gmail.com> Michael Foord wrote: > I agree with Martin that the *perception* is that to use Python 2.6 to > help you port to Python 3 you have to be willing to drop support for > earlier versions of Python. Note that increased 3.x compatibility in the most recent 2.x release will always help in two scenarios: 1. New projects that want to use 2.x only libraries but want to be ready for the Py3k transition in their own code (e.g. new 2.7 features like set literals, dict and set comprehensions and the view* dict methods can be translated far more cleanly by 2to3 than the closest comparable 2.6 code). 2. Similarly new projects that use a 3.x trunk can be more readily translated to a 2.7 version with 3to2, whereas some constructs may be difficult to recreate in earlier Python versions. I would expect this to be significantly less common then the first scenario though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:38:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:38:31 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4DA2A7.2080408@gmail.com> Antoine Pitrou wrote: > Christian Heimes cheimes.de> writes: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) > > +1. I don't care about trademarks or official names, we should call it whatever > is obvious for our users. Until this discussion, I didn't even know the architecture had any name other than x86-64. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:39:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:39:40 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4D36AA.7020607@palladion.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> Message-ID: <4B4DA2EC.3050908@gmail.com> Tres Seaver wrote: > There is an obnoxious deprecation warning out of the distutils: > > DeprecationWarning: 'compiler' specifies the compiler type in > build_ext. If you want to get the compiler object itself, use > 'compiler_obj' > > which is likely a simple one-line fix, if I only knew what the hell it > was whining about. ;) The warning is extra obnoxious because it doesn't > tell me what in *my* code triggers the warning (it (needs a 'stacklevel' > argument). Could you kick a tracker issues in Tarek's direction for that one? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Wed Jan 13 13:24:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 13 Jan 2010 12:24:09 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Brett Cannon python.org> writes: > If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. I'm sorry I won't be able to come to the language summit, but I would like if possible to expedite a pronouncement on PEP 391 (configuring logging using dictionaries). I believe I addressed all the comments made on the discussion threads mentioned in the PEP and so I'm not sure what more I need to do to get a pronouncement. I guess the stdlib slot gives an opportunity for people to air their views and so I'd be grateful if you added it to the agenda. Thanks & regards, Vinay Sajip From murman at gmail.com Wed Jan 13 14:35:51 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 07:35:51 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. I see and agree with the motivation behind your point, but it's just as reasonable to mimic the name Microsoft uses: the release is for Windows rather than the processor. On MSDN Microsoft uses parentheticals x86, ia64 and x64; this would suggest a name like Python 2.6.4 installer for Windows (x64). Unfortunately this usage doesn't seem to be reflected in consumer-oriented product pages, so I'm uncertain how clear it is for those downloading Python. -- Michael Urman From ncoghlan at gmail.com Wed Jan 13 15:04:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 00:04:28 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4DD2EC.3030709@gmail.com> Michael Urman wrote: > On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. > > I see and agree with the motivation behind your point, but it's just > as reasonable to mimic the name Microsoft uses: the release is for > Windows rather than the processor. On MSDN Microsoft uses > parentheticals x86, ia64 and x64; this would suggest a name like > Python 2.6.4 installer for Windows (x64). Unfortunately this usage > doesn't seem to be reflected in consumer-oriented product pages, so > I'm uncertain how clear it is for those downloading Python. As Windows doesn't run on non-x86 architectures, their naming is generally just Windows (32 bit) and Windows (64 bit). Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. In this case, making it clear that the 64-bit Windows binaries work for both Intel and AMD chips is more important than using only the official architecture name. Cheers, Nick. P.S. This discussion made me realise that out of habit I had dropped the 32-bit version of Python on my new 64-bit Windows box... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Wed Jan 13 17:49:55 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 13 Jan 2010 11:49:55 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA2EC.3050908@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> <4B4DA2EC.3050908@gmail.com> Message-ID: <4B4DF9B3.4030403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Tres Seaver wrote: >> There is an obnoxious deprecation warning out of the distutils: >> >> DeprecationWarning: 'compiler' specifies the compiler type in >> build_ext. If you want to get the compiler object itself, use >> 'compiler_obj' >> >> which is likely a simple one-line fix, if I only knew what the hell it >> was whining about. ;) The warning is extra obnoxious because it doesn't >> tell me what in *my* code triggers the warning (it (needs a 'stacklevel' >> argument). > > Could you kick a tracker issues in Tarek's direction for that one? Done: http://bugs.python.org/issue7694 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktN+bMACgkQ+gerLs4ltQ6s4gCdENhtVQWzoH449BCDz8SNx/4s DEoAn19LMcMs3b6ctdNF8K2hYd6d+BSZ =TWit -----END PGP SIGNATURE----- From rdmurray at bitdance.com Wed Jan 13 18:24:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Jan 2010 12:24:42 -0500 Subject: [Python-Dev] PYTHON3PATH Message-ID: <20100113172442.4A7771FA679@kimball.webabinitio.net> Please review issue 2375 [1], which is an enhancement request to add a PYTHON3PATH environment variable. Because we have elected to have both a python and a python3 command, I think this is an issue worth thinking about carefully to make sure we are serving the Python user community and easing the transition to python3. It could be that "use virtualenv" is the best answer, but I feel we should think about it carefully to make sure that is really true. [1] http://bugs.python.org/issue2375 -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From guido at python.org Wed Jan 13 18:31:39 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:31:39 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: Somehow the bug site doesn't load for me right now, but I'm -1 on this. There are maybe a dozen PYTHON... variables -- we really shouldn't try to add PYTHON3 variants for all of them. Specifically for PYTHONPATH, I feel that its use is always a short-time or localized hack, not something you set in your .profile nd forget about it (forgetting about it inparticular often leads to unnecessary debugging pain later). The better approaches are based on site-packages, wrapper scripts, virtualenv, etc. --Guido On Wed, Jan 13, 2010 at 9:24 AM, R. David Murray wrote: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. ?Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. ?It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 > > -- > R. David Murray ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dirkjan at ochtman.nl Wed Jan 13 18:38:26 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 13 Jan 2010 18:38:26 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 18:29, Benjamin Peterson wrote: > Please consider trying Python 2.7 with your code and reporting any bugs you may I ran the Mercurial test suite on current trunk, and it worked alright. There was a small issue with the fact that mimetypes got fooled by the lack of ImportError from _winreg (which I solved by blacklisting _winreg in demandimport), and one test fails likely because of a change in MIME handling. Will look into that. So the state of trunk looks rather solid from where I sit. Cheers, Dirkjan From ralf at brainbot.com Wed Jan 13 18:40:04 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:40:04 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> (R. David Murray's message of "Wed, 13 Jan 2010 12:24:42 -0500") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87r5ptdhu3.fsf@muni.brainbot.com> "R. David Murray" writes: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 The first thing I got while trying to run a python3 prompt few days ago, was an error. python3 tried to read my $PYTHONSTARTUP file, which used print statements. people will have to run both python 2 and python 3 code at the same time. Using different environment variables will make this easier. - Ralf From ralf at brainbot.com Wed Jan 13 18:55:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:55:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Guido van Rossum's message of "Wed, 13 Jan 2010 09:31:39 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87k4vldh4c.fsf@muni.brainbot.com> Guido van Rossum writes: > Somehow the bug site doesn't load for me right now, but I'm -1 on > this. There are maybe a dozen PYTHON... variables -- we really > shouldn't try to add PYTHON3 variants for all of them. > > Specifically for PYTHONPATH, I feel that its use is always a > short-time or localized hack, not something you set in your .profile > nd forget about it (forgetting about it inparticular often leads to > unnecessary debugging pain later). The better approaches are based on > site-packages, wrapper scripts, virtualenv, etc. then at least remove them completely from python3, so one can use them for his python 2 installation. Otherwise it will stump on some innocent python 3 program (and cause unnecessary debugging pain). From steven.bethard at gmail.com Wed Jan 13 18:57:29 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 13 Jan 2010 09:57:29 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: On Wed, Jan 13, 2010 at 9:40 AM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. ?Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. ?It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. How complicated is your PYTHONSTARTUP file? My suspicion is that you could easily write it to work for both Python 2.X and 3.X. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ssteinerx at gmail.com Wed Jan 13 18:57:42 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 13 Jan 2010 12:57:42 -0500 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> On Jan 13, 2010, at 12:40 PM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. Python 2 will continue to run as it has, and better solutions will emerge for Python 3 development. Beats the heck out of making a whole duplicate set for PYTHON3BLAHBLAH, yuck! S From guido at python.org Wed Jan 13 18:58:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:58:04 -0800 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: Memories of days past... Python had several regular expression implementations before, one of which was called "regex". But I would rather not have a new module -- I would much rather have a flag specifying the new (backwards incompatible) syntax/semantics. The flag would have a long name (e.g. re.NEW_SYNTAX), a short name (e.g. re.N) and an inline syntax, "(?n)...". --Guido On Tue, Jan 12, 2010 at 7:58 PM, Brett Cannon wrote: > > > On Tue, Jan 12, 2010 at 14:10, MRAB wrote: >> >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. >> > > If it is a separate module under a different name it can do the proper > thing. People will just need to be aware of the difference when they import > the module. > >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? >> > > That's totally up to you. There is practically no chance of it getting into > the 2.x under the stdlib at this point since 2.7b1 is coming up and this > module has not been out in the wild for a year (to my knowledge). ?If you > want to support 2.x that's fine and I am sure users would appreciate it, but > it isn't necessary to get into the Python 3 stdlib. > >> >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? >> > > If you want to minimize the differences then it should probably match. As I > said, since it is a different name to import under it can deviate where > reasonable, just make sure to clearly document the deviations. > -Brett > >> >> Thanks >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From ralf at brainbot.com Wed Jan 13 19:03:08 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 19:03:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Steven Bethard's message of "Wed, 13 Jan 2010 09:57:29 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <87fx69dgrn.fsf@muni.brainbot.com> Steven Bethard writes: > > How complicated is your PYTHONSTARTUP file? My suspicion is that you > could easily write it to work for both Python 2.X and 3.X. sure. that's exactly what I did. My point is that sharing those environment variables will cause pain for some people. From guido at python.org Wed Jan 13 19:07:58 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:07:58 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. -1. They have their use, but more in controlled situations. If you have "global" env vars that you only want to use with Python 2.x, write a wrapper for Python 3 that invokes it with -E. -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Wed Jan 13 19:14:21 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 13 Jan 2010 13:14:21 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E0D7D.3040806@scottdial.com> On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Wed Jan 13 19:22:33 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:22:33 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Wed, Jan 13, 2010 at 4:24 AM, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> If there is something missing from the stdlib discussion that you think should > be brought up at the summit let me know. And if there is something here you want > to discuss before the summit let me know and I can start a separate thread on it. > > I'm sorry I won't be able to come to the language summit, but I would like if > possible to expedite a pronouncement on PEP 391 (configuring logging using > dictionaries). I believe I addressed all the comments made on the discussion > threads mentioned in the PEP and so I'm not sure what more I need to do to get a > pronouncement. I guess the stdlib slot gives an opportunity for people to air > their views and so I'd be grateful if you added it to the agenda. Is the PEP in the stage of consensus yet? If so it may not even be necessary to discuss it at the summit (though I certainly won't stop people if they want to). -- --Guido van Rossum (python.org/~guido) From phd at phd.pp.ru Wed Jan 13 19:18:50 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 21:18:50 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <20100113181850.GA3837@phd.pp.ru> On Wed, Jan 13, 2010 at 12:57:42PM -0500, ssteinerX at gmail.com wrote: > Or, how about just removing the antiquated use of environment variables "antiquated"? You are kidding, aren't you?! Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Jan 13 19:51:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:51:14 -0800 Subject: [Python-Dev] PyCon Keynote Message-ID: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Jan 13 20:13:58 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:13:58 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D9310.6060907@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> Message-ID: <4B4E1B76.4010309@v.loewis.de> >>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>> X86-64 binary -- does not include source) >>> >>> instead of: >>> >>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>> not include source) >>> >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. >> > > I think we should use whatever is most informative and least confusing > to our users - we owe our allegiance to them and not to a processor vendor. And why do you think this is x86-64? Regards, Martin From martin at v.loewis.de Wed Jan 13 20:33:24 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:33:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA0DA.7070007@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> <4B4DA0DA.7070007@gmail.com> Message-ID: <4B4E2004.8050905@v.loewis.de> > Note that increased 3.x compatibility in the most recent 2.x release > will always help in two scenarios: > > 1. New projects that want to use 2.x only libraries but want to be ready > for the Py3k transition in their own code (e.g. new 2.7 features like > set literals, dict and set comprehensions and the view* dict methods can > be translated far more cleanly by 2to3 than the closest comparable 2.6 > code). Of these, I can only see this being the case of the view* methods. Why would set literals allow for code that more cleanly translates to 3.x? Suppose you write f = set((1,2,3)) in 2.6. That code will work *exactly* the same way in 3.1, no translation needed at all. Likewise for dict and set comprehensions: the code is as cleanly translated in the 2.7 form as it is in any prior form (ie. no modifications). As for view* methods: there is one important exception where the 2.6 equivalent is as cleanly translated as a view method: for key in D: action This is a more modern equivalent for iterating over D.keys(), so if your code still does the latter, just change it to the former (requires Python 2.2). I'd claim that this is the dominant case of traversing a dictionary (probably followed by iterating over .items()). So while it is true that only view* can be transformed correctly, I'd claim that this is an infrequent issue. > 2. Similarly new projects that use a 3.x trunk can be more readily > translated to a 2.7 version with 3to2, whereas some constructs may be > difficult to recreate in earlier Python versions. That may be true, but is besides the point here: the issue was whether a 2.8 release might help in migrating to 3.x. People who follow this approach already have 3.x code. Whether they would rather port to 2.8 only, and get that work with little effort, or whether they would port to 2.5 and later, and put in more work, I don't know - I guess we would have to ask these people. I would expect that they prefer if 2.x died rather sooner than later, because they can then stop supporting it. Regards, Martin From tjreedy at udel.edu Wed Jan 13 20:36:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Jan 2010 14:36:03 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: On 1/13/2010 2:13 PM, "Martin v. L?wis" wrote: >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. > > And why do you think this is x86-64? I do not believe I have personally seen, or at least noticed, that specific term before. Something like "Release for 64-bit Windows (Win NT, 2000, XP, Vista, 7 ) on AMD64-type processors from AMD and Intel" should be clear enough. From martin at v.loewis.de Wed Jan 13 20:40:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 20:40:30 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E21AE.40602@v.loewis.de> > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). Windows actually does - it runs on IA-64 (which is a non-x86 architecture). However, end users are unlikely to use such hardware, so distinguishing between 32-bit and 64-bit is typically good enough. > In this case, making it clear that the 64-bit Windows binaries work for > both Intel and AMD chips is more important than using only the official > architecture name. Well, we did have Itanium binaries at one point, and the "64-bit Windows binaries" you are referring to most definitely don't run on Itanium, even though that's an Intel chip. Regards, Martin From martin at v.loewis.de Wed Jan 13 20:45:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:45:40 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E0D7D.3040806@scottdial.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> Message-ID: <4B4E22E4.4040504@v.loewis.de> > So to echo what Michael said, the Microsoft nomenclature is "x64" > regardless of yours and Martin's objections to that name. Nobody who > uses Windows would be confused by "x64" since that is *the* Microsoft > naming scheme. That's actually not entirely true. There are several places in the APIs where Microsoft either allows or requires to call the architecture AMD64 (e.g. architecture indication in MSI files). Regards, Martin From regebro at gmail.com Wed Jan 13 20:50:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 20:50:59 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. What do you need to do in the PYTHONSTARTUP file? Ten years of Python programming, and I didn't even know it existed. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From phd at phd.pp.ru Wed Jan 13 21:08:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 23:08:40 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <20100113200840.GC14858@phd.pp.ru> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From murman at gmail.com Wed Jan 13 21:12:11 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 14:12:11 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E22E4.4040504@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> <4B4E22E4.4040504@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 13:45, "Martin v. L?wis" wrote: >> So to echo what Michael said, the Microsoft nomenclature is "x64" >> regardless of yours and Martin's objections to that name. Nobody who >> uses Windows would be confused by "x64" since that is *the* Microsoft >> naming scheme. > > That's actually not entirely true. There are several places in the > APIs where Microsoft either allows or requires to call the architecture > AMD64 (e.g. architecture indication in MSI files). I should have clarified I was talking about the names shown on MSDN subscriptions for downloading installation media of Windows 7 and Windows Vista. It's pretty clear in that context Microsoft uses x64 to describe this platform of Windows. But again, it's far from clear that this is a term they use for non-developers. -- Michael Urman From regebro at gmail.com Wed Jan 13 21:27:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 21:27:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113200840.GC14858@phd.pp.ru> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> Message-ID: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: > On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >> What do you need to do in the PYTHONSTARTUP file? >> Ten years of Python programming, and I didn't even know it existed. :-) > > ? See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Cheese and fries! :-) Anyway, I don't see how this could pose any problems, because any user advanced enough to do these things will be advanced enough to understand what the problem is and fix it so it works under Python 3 too. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ralf at brainbot.com Wed Jan 13 21:52:34 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 20:52:34 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> (Lennart Regebro's message of "Wed, 13 Jan 2010 20:50:59 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <874ompg225.fsf@brainbot.com> Lennart Regebro writes: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) hehe. tab completion: # -*- mode: python -*- # Last changed: 2009-12-23 22:25:15 by ralf import sys import os def initreadline(): try: import readline except ImportError: sys.stdout.write("Module readline not available.\n") return import rlcompleter readline.parse_and_bind("tab: complete") # Use tab for completions readline.parse_and_bind('tab: complete') # This forces readline to automatically print the above list when tab # completion is set to 'complete'. readline.parse_and_bind('set show-all-if-ambiguous on') # Bindings for incremental searches in the history. These searches # use the string typed so far on the command line and search # anything in the previous input history containing them. readline.parse_and_bind('"\C-r": reverse-search-history') readline.parse_and_bind('"\C-s": forward-search-history') history = os.path.expanduser("~/.pyhistory") if os.path.exists(history): readline.read_history_file(history) import atexit atexit.register(lambda: readline.write_history_file(history)) initreadline() del initreadline From martin at v.loewis.de Wed Jan 13 22:05:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 22:05:03 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <4B4E357F.4050107@v.loewis.de> Lennart Regebro wrote: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? I personally set up readline: set tab completion, and load the history of commands from the previous session. Of course, I don't know what Ralf uses it for. Regards, Martin From ncoghlan at gmail.com Wed Jan 13 22:43:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 07:43:41 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <4B4E3E8D.2010407@gmail.com> Guido van Rossum wrote: > On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about > just removing the antiquated use of environment variables altogether > from Python 3 and avoid the issue completely. > > -1. They have their use, but more in controlled situations. If you > have "global" env vars that you only want to use with Python 2.x, > write a wrapper for Python 3 that invokes it with -E. Perhaps a case can be made for Python 3 to assume -E by default, with a -e option to enable reading of the environment variables? That way naive users could run Python3 without tripping over existing Python2 environment variables, while other tools could readily set up a different environment before launching Python3. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Wed Jan 13 23:45:57 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 14:45:57 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 1:43 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users using environment variables? That's a recipe for disaster in any version! :-) -- --Guido van Rossum (python.org/~guido) From jared.grubb at gmail.com Wed Jan 13 23:56:37 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Wed, 13 Jan 2010 14:56:37 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <13F6C43A-D62A-4A9F-AF7A-9BDAC94F7D72@gmail.com> On 13 Jan 2010, at 13:43, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. I raised a question about these PYTHON3* variables once before in a discussion about shebang lines: http://www.mailinglistarchive.com/python-dev at python.org/msg29500.html I'm not advocating them, but just wanted to make sure to bring up the shebang use case. Jared From skip at pobox.com Wed Jan 13 23:24:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 16:24:13 -0600 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <19278.18445.428716.321365@montanaro.dyndns.org> Lennart> What do you need to do in the PYTHONSTARTUP file? Just reading off stuff from my own personal startup file... I use it for stuff I want available during interactive sessions: 1. Enable true division. 2. Conditionally define "help" from back in the days when there was no help builtin function. 3. Auto-save session (readline) history so I can easily recall commands across sessions. 4. Add other fake builtins ("like") or override behavior of some (like "dir") making them handier for interactive use. 5. autoload modules/symbols (pokes around in common modules from sys.excepthook function). Oh, and I've had no particular trouble keeping it working in Python 1, 2 or 3. Skip From fuzzyman at voidspace.org.uk Thu Jan 14 01:20:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 00:20:21 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: <4B4E6345.5070202@voidspace.org.uk> On 13/01/2010 19:13, "Martin v. L?wis" wrote: >>>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>>> X86-64 binary -- does not include source) >>>> >>>> instead of: >>>> >>>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>>> not include source) >>>> >>>> >>> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >>> to use AMD64 instead. I think we should comply - they invented the >>> architecture, so they have the right to give it a name. Neither >>> Microsoft nor Intel have such a right. >>> >>> >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. >> > And why do you think this is x86-64? > Well anecdotal everyone I have *every* talked to about 64bit processors has referred to having a 64bit processor (x86 is a given) and not an AMD64 architecture processor. Linus Torvalds addressed this specific issue for Linux and came down on the side of "x86-64": http://kerneltrap.org/node/2466 Look up AMD64 on Wikipedia and it redirects you to the X86-64 page. Information website setup by AMD and partners about the AMD64 architecture: http://www.x86-64.org/about.html In the AMD website they refer to "x86-64 Assembly": http://www.x86-64.org/documentation/assembly.html Microsoft seem to draw a distinction between x64 (which would also be acceptable) and Itanium based systems. Very rarely do they refer to AMD64: * http://www.microsoft.com/servers/64bit/compare.mspx * http://www.microsoft.com/servers/64bit/x64/overview.mspx * http://www.microsoft.com/servers/64bit/overview.mspx Using a vendor specific name automatically begs the question as to whether the installer works on processors from other vendors, as we saw in the specific enquiry from the user that triggered this debate. Referring to the AMD 64 build as x86-64, with a footnote explaining which architectures this specifically means is unlikely to confuse people. It is *definitely* better than just saying AMD64. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Thu Jan 14 02:50:55 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 19:50:55 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? Message-ID: <19278.30847.649228.115514@montanaro.dyndns.org> Just out of curiosity, is the static build stuff (use the old Modules/Setup file to build modules) exercised at all any more? Skip From benjamin at python.org Thu Jan 14 03:22:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 13 Jan 2010 20:22:03 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? In-Reply-To: <19278.30847.649228.115514@montanaro.dyndns.org> References: <19278.30847.649228.115514@montanaro.dyndns.org> Message-ID: <1afaf6161001131822r151bd202x35653ba44ee0235d@mail.gmail.com> 2010/1/13 : > > Just out of curiosity, is the static build stuff (use the old Modules/Setup > file to build modules) exercised at all any more? Exercised as in used in the build process? Yes. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Jan 14 10:23:47 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 09:23:47 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! Message-ID: <954450.26617.qm@web25804.mail.ukl.yahoo.com> In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: http://www.python.org/dev/peps/pep-0391/ In November 2009 I posted to this list that the PEP was ready for review. I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. Thanks & regards, Vinay Sajip From ziade.tarek at gmail.com Thu Jan 14 10:30:15 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Jan 2010 10:30:15 +0100 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <94bdd2611001140130u6154e893jfd37db32a25be66a@mail.gmail.com> On Thu, Jan 14, 2010 at 10:23 AM, Vinay Sajip wrote: [..] > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would > be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check > in the changes unless there are objections to doing so. All those who think that logging is currently > hard to configure will benefit from these changes, so here's the opportunity to pipe up and > improve things. FWIW, I am +1. Thanks for your work Tarek From hodgestar+pythondev at gmail.com Thu Jan 14 10:39:41 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 14 Jan 2010 11:39:41 +0200 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: On Thu, Jan 14, 2010 at 11:23 AM, Vinay Sajip wrote: > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I think the dict configuration scheme will be a great addition to the logging module. :) Schiavo Simon From p.f.moore at gmail.com Thu Jan 14 12:22:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jan 2010 11:22:09 +0000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> 2010/1/14 Vinay Sajip : > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I've no immediate need for the feature, but it would be good to have something like this, so I'm +1. Paul. From ncoghlan at gmail.com Thu Jan 14 12:46:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:46:19 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> Message-ID: <4B4F040B.8010607@gmail.com> Paul Moore wrote: > 2010/1/14 Vinay Sajip : >> So, can you please indicate your vote for or against incorporating PEP 391 into Python? > > I've no immediate need for the feature, but it would be good to have > something like this, so I'm +1. I'm in the same boat as Paul, but PEP 291 provides a nice forward compatible configuration scheme that will work with any application configuration approach that can produce an appropriate Python dictionary. So +1 from me too - I think the PEP has now taken this as far as theorising can go, and the only way to learn anything further is to put it into practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jan 14 12:57:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:57:55 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4B4F06C3.7030806@gmail.com> Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of > configuring logging using dictionaries: > > http://www.python.org/dev/peps/pep-0391/ Although one minor comment: you can probably remove the note about the "ext://" and "cfg://" prefixes being provisional at this stage. Those names look fine to me, so I don't see any point inviting a late-breaking bikeshed discussion about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu Jan 14 14:53:29 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 08:53:29 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> On Thu, Jan 14, 2010 at 4:23 AM, Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: > > ?http://www.python.org/dev/peps/pep-0391/ > > In November 2009 I posted to this list that the PEP was ready for review. > > I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. > > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. > > Thanks & regards, > > Vinay Sajip I'm generally +1 - but given I know that Django 1.2 is slated to implement something somewhat similar, I'm interested to hear how this proposal meshes with their plan(s). jesse From vinay_sajip at yahoo.co.uk Thu Jan 14 15:08:24 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 14:08:24 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> Message-ID: <92210.91656.qm@web25806.mail.ukl.yahoo.com> > From: Jesse Noller > I'm generally +1 - but given I know that Django 1.2 is slated to > implement something somewhat similar, I'm interested to hear how this > proposal meshes with their plan(s).. Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: http://groups.google.com/group/django-developers/msg/4ef81a2160257221 They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. Regards, Vinay Sajip From ssteinerx at gmail.com Thu Jan 14 15:54:27 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 14 Jan 2010 09:54:27 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <62E362ED-5DD7-4D42-B5E0-B263A137FD5C@gmail.com> On Jan 14, 2010, at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. That puts a huge +1 on there for me. If we can get this approved and have Django's logging improved *and* avoid a whole pile of duplicate effort in Django that's a huge win. S From jnoller at gmail.com Thu Jan 14 15:56:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 09:56:18 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <4222a8491001140656w68c7290agccfe7282cf96f2fb@mail.gmail.com> On Thu, Jan 14, 2010 at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. > > Regards, > > Vinay Sajip > Cool, +1 then :) From mal at egenix.com Thu Jan 14 20:19:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 20:19:12 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <4B4F6E30.50400@egenix.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users won't have PYTHONPATH or any other Python environment variables setup. Really, if you know that you are going to run Python 3 instead of Python 2 or vice-versa it's easy enough to run . py3env.sh or . py2env.sh in order to setup your shell environment, much like you typically do when using virtual Python installations or work on different projects that require different setups. If you just want to separate Python 2 and 3 files, use the user site-packages dir which includes the Python version. More experienced users could also write their own environment switching sitecustomize.py or usercustomize.py. And then there's the hackery option for experts that love dirty tricks: add a .pth file which includes an "import mysetup" line to fix-up you path and other settings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jan 14 22:02:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Jan 2010 07:02:28 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <4B4F8664.4080107@gmail.com> M.-A. Lemburg wrote: > Nick Coghlan wrote: >> Guido van Rossum wrote: >>> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >>> just removing the antiquated use of environment variables altogether >>> from Python 3 and avoid the issue completely. >>> >>> -1. They have their use, but more in controlled situations. If you >>> have "global" env vars that you only want to use with Python 2.x, >>> write a wrapper for Python 3 that invokes it with -E. >> Perhaps a case can be made for Python 3 to assume -E by default, with a >> -e option to enable reading of the environment variables? >> >> That way naive users could run Python3 without tripping over existing >> Python2 environment variables, while other tools could readily set up a >> different environment before launching Python3. > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. I was actually thinking of the situation where the OS had a preinstalled Python 2 with a default PYTHONPATH setting and the user was playing with Python 3. However, I agree that that is a fairly unlikely scenario (since preinstalled Pythons tend not to rely on the environment variables and a user sophisticated enough to be playing with a new Python interpreter should be able to cope with a few environment variable conflicts anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Thu Jan 14 22:09:30 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 21:09:30 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F8664.4080107@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <4B4F8664.4080107@gmail.com> Message-ID: <4B4F880A.5010809@voidspace.org.uk> On 14/01/2010 21:02, Nick Coghlan wrote: > However, I agree that that is a fairly unlikely scenario (since > preinstalled Pythons tend not to rely on the e Well, on the other hand I think that during the next few years it will be increasingly common for developers (and possibly users) to have Python 2 and Python 3 installed side-by-side. Many libraries and applications may never make the jump to Python 3 and Python users may be using 'legacy' Python 2 code for many years to come. It will also become increasingly common for developers to be using Python 3 *primarily* and for Python 3 only libraries and applications to emerge. Whilst there are workarounds we *are* in a situation that Python 2 and Python 3 share environment variables for the location of libraries and executing code on startup, whilst at the same time they are largely incompatible and need separate library paths and startup code. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ralf at brainbot.com Thu Jan 14 22:25:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Thu, 14 Jan 2010 22:25:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> (M.'s message of "Thu, 14 Jan 2010 20:19:12 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <871vhswf90.fsf@brainbot.com> "M.-A. Lemburg" writes: > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. > > Really, if you know that you are going to run Python 3 instead of > Python 2 or vice-versa it's easy enough to run You don't even know that you're going to run python. I have 40 python scripts in my /usr/bin directory. > > . py3env.sh > or > . py2env.sh > > in order to setup your shell environment, much like you typically > do when using virtual Python installations or work on different > projects that require different setups. No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > > If you just want to separate Python 2 and 3 files, use the > user site-packages dir which includes the Python version. Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to install those programs in a user's home directory. There are still people running python <2.6. From mal at egenix.com Thu Jan 14 22:51:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 22:51:07 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <871vhswf90.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <871vhswf90.fsf@brainbot.com> Message-ID: <4B4F91CB.2060106@egenix.com> Ralf Schmitt wrote: > "M.-A. Lemburg" writes: > >> >> Naive users won't have PYTHONPATH or any other Python environment >> variables setup. >> >> Really, if you know that you are going to run Python 3 instead of >> Python 2 or vice-versa it's easy enough to run > > You don't even know that you're going to run python. I have 40 python > scripts in my /usr/bin directory. > >> >> . py3env.sh >> or >> . py2env.sh >> >> in order to setup your shell environment, much like you typically >> do when using virtual Python installations or work on different >> projects that require different setups. > > No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > >> >> If you just want to separate Python 2 and 3 files, use the >> user site-packages dir which includes the Python version. > > Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to > install those programs in a user's home directory. There are still > people running python <2.6. Note that you only need to have a different PYTHONPATH setups for Python 2.x/3.x if you plan to run code that: * is not installed in site-packages (most OS shipped code will be found in the resp. system site-packages/ dir) * is not installed in a user site-packages directory (user installed code will typically go there (*)) * uses modules/packages that come in two different versions (one for Python 2.x and one for 3.x) and use the same name for both versions * needs to work in both Python 2.x and 3.x (*) The method for installing code in user site-packages dir is running: python setup.py install --user instead of the standard python setup.py install which install to the system-wide site-packages. BTW: I guess the bzr and mercurial wikis will need to be updated accordingly - at least for users of Python >=2.6. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Jan 14 22:55:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jan 2010 22:55:04 +0100 Subject: [Python-Dev] [ANN] Python 2.5.5 Release Candidate 1. Message-ID: <4B4F92B8.30806@v.loewis.de> On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.5. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.6 (which is 2.6.4 at this point). This releases fixes issues with the logging and tarfile modules, and with thread-local variables. See the detailed release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed. For more information on Python 2.5.5, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.5.5 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin at v.loewis.de Python Release Manager (on behalf of the entire python-dev team) From status at bugs.python.org Fri Jan 15 18:07:26 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Jan 2010 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100115170726.9963A7865F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/08/10 - 01/15/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2536 open (+32) / 16993 closed (+16) / 19529 total (+48) Open issues with patches: 1024 Average duration of open issues: 710 days. Median duration of open issues: 469 days. Open Issues Breakdown open 2501 (+32) pending 34 ( +0) Issues Created Or Reopened (53) _______________________________ PYTHON3PATH environment variable to supersede PYTHONPATH for mul 01/13/10 CLOSED http://bugs.python.org/issue2375 reopened r.david.murray Mark the compiler package as deprecated 01/13/10 http://bugs.python.org/issue6837 reopened ezio.melotti test_distutils failure 01/15/10 CLOSED http://bugs.python.org/issue6961 reopened flox buildbot test_urllib: unsetting missing 'env' variable 01/08/10 CLOSED http://bugs.python.org/issue7026 reopened flox Two float('nan') are not equal 01/08/10 CLOSED http://bugs.python.org/issue7660 created jmfauth compiling ctypes fails with non-ascii path 01/08/10 http://bugs.python.org/issue7661 created pitrou patch time.utcoffset() 01/09/10 http://bugs.python.org/issue7662 created techtonik UCS4 build incorrectly translates cases for non-BMP code points 01/10/10 CLOSED http://bugs.python.org/issue7663 created exarkun python logger does not handle IOError Exception 01/10/10 CLOSED http://bugs.python.org/issue7664 created yateenjoshi test_urllib2 and test_ntpath fail if path contains "\" 01/10/10 http://bugs.python.org/issue7665 created flox test_lib2to3 fails if path contains space 01/10/10 CLOSED http://bugs.python.org/issue7666 created flox test_doctest fails with non-ascii path 01/10/10 http://bugs.python.org/issue7667 created flox buildbot test_httpservers fails with non-ascii path 01/10/10 http://bugs.python.org/issue7668 created flox buildbot test_unicode_file fails with non-ascii path 01/10/10 CLOSED http://bugs.python.org/issue7669 created flox _sqlite3: Block *all* operations on a closed Connection object 01/10/10 http://bugs.python.org/issue7670 created haypo patch test_popen fails if path contains special char like ";" 01/11/10 http://bugs.python.org/issue7671 reopened flox patch _ssl module causes segfault 01/10/10 http://bugs.python.org/issue7672 created ssoria patch audioop: check that length is a multiple of the size 01/11/10 http://bugs.python.org/issue7673 created haypo patch select.select() corner cases: duplicate fds, out-of-range fds 01/11/10 http://bugs.python.org/issue7674 created cdleary help() doesn't accept unicode literals in built in docstrings 01/11/10 http://bugs.python.org/issue7675 created psd patch IDLE shell shouldn't use TABs 01/11/10 http://bugs.python.org/issue7676 created cben distutils, better error message for setup.py upload -sign withou 01/11/10 http://bugs.python.org/issue7677 created illume subprocess.Popen pipeline example code in the documentation is l 01/11/10 http://bugs.python.org/issue7678 created steven.k.wong Warning building 2.7 on OS X 10.6 libintl.h "Present But Cannot 01/12/10 http://bugs.python.org/issue7679 created ssteinerX pythonw crash while attempting to start() a thread object 01/12/10 http://bugs.python.org/issue7680 created dontbugme Incorrect division in [wave.py] 01/12/10 CLOSED http://bugs.python.org/issue7681 created alfps patch, needs review Optimisation of if with constant expression 01/12/10 http://bugs.python.org/issue7682 created sdefresne patch Wrong link in HTML documentation 01/12/10 CLOSED http://bugs.python.org/issue7683 created francescor decimal.py: infinity coefficients in tuples 01/12/10 http://bugs.python.org/issue7684 created skrah minor typo in re docs 01/12/10 CLOSED http://bugs.python.org/issue7685 created mikejs patch redundant open modes 'rbb', 'wbb', 'abb' no longer work on Windo 01/13/10 http://bugs.python.org/issue7686 created ivank Bluetooth support untested 01/13/10 http://bugs.python.org/issue7687 created pitrou TypeError: __name__ must be set to a string object 01/13/10 http://bugs.python.org/issue7688 created frankmillman Pickling of classes with a metaclass and copy_reg 01/13/10 http://bugs.python.org/issue7689 created craigcitro patch Wrong PEP number in importlib module manual page 01/13/10 CLOSED http://bugs.python.org/issue7690 created francescor test_bsddb3 files are not always removed when test fails 01/13/10 CLOSED http://bugs.python.org/issue7691 created flox buildbot Pointless comparision of unsigned integer with zero 01/13/10 CLOSED http://bugs.python.org/issue7692 created Bugger tarfile.extractall can't have unicode extraction path 01/13/10 http://bugs.python.org/issue7693 created pbienst DeprecationWarning from distuils.commands.build_ext needs stackl 01/13/10 http://bugs.python.org/issue7694 created tseaver patch missing termios constants 01/13/10 http://bugs.python.org/issue7695 created conorh Improve Memoryview/Buffer documentation 01/13/10 http://bugs.python.org/issue7696 created flox os.getcwd() should raise UnicodeDecodeError for arbitrary bytes 01/13/10 CLOSED http://bugs.python.org/issue7697 created flox pystack macro in Misc/gdbinit incorrectly uses PyEval_EvalFrame 01/14/10 CLOSED http://bugs.python.org/issue7698 created exarkun patch strptime, strftime documentation 01/14/10 http://bugs.python.org/issue7699 created mikejs patch "-3" flag does not work anymore 01/14/10 CLOSED http://bugs.python.org/issue7700 created flox patch fix output string length for binascii.b2a_uu() 01/14/10 CLOSED http://bugs.python.org/issue7701 created haypo patch Wrong order of parameters of _get_socket in SMTP class in smtpli 01/14/10 http://bugs.python.org/issue7702 created alito Replace buffer()-->memoryview() in Lib/ctypes/test/ 01/14/10 http://bugs.python.org/issue7703 created flox patch Math calculation problem (1.6-1.0)>0.6, python said TRUE 01/14/10 CLOSED http://bugs.python.org/issue7704 created DhaReaL libpython2.6.so is not linked correctly on FreeBSD when threads 01/15/10 http://bugs.python.org/issue7705 created aballier patch, needs review Missing #include guards 01/15/10 http://bugs.python.org/issue7706 created akrpic77 patch multiprocess.Queue operations during import can lead to deadlock 01/15/10 http://bugs.python.org/issue7707 created kripken Conversion of longs to bytes and vice-versa. 01/11/10 http://bugs.python.org/issue1023290 reopened mark.dickinson patch Issues Now Closed (67) ______________________ segfault in curses when calling redrawwin() before refresh() 825 days http://bugs.python.org/issue1266 mark.dickinson "RuntimeError: dictionary changed size during iteration" in Tkin 751 days http://bugs.python.org/issue1658 flox patch Exception exceptions.AttributeError '_shutdown' in 0.6, python said TRUE 0 days http://bugs.python.org/issue7704 tim_one Support for sending multipart form data 2452 days http://bugs.python.org/issue727898 r.david.murray email.MIME*.as_string removes duplicate spaces 1440 days http://bugs.python.org/issue1422094 r.david.murray easy test_parsedate_acceptable_to_time_functions+DST == :-( 1394 days http://bugs.python.org/issue1454285 r.david.murray email module does not complay with RFC 2046: CRLF issue 1196 days http://bugs.python.org/issue1571841 r.david.murray email.FeedParser.BufferedSubFile improperly handles "\r\n" 969 days http://bugs.python.org/issue1721862 r.david.murray patch, easy Top Issues Most Discussed (10) ______________________________ 20 dtoa.c: oversize b in quorem 11 days open http://bugs.python.org/issue7632 18 _ssl module causes segfault 5 days open http://bugs.python.org/issue7672 12 Speed up cPickle's pickling generally 287 days open http://bugs.python.org/issue5683 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 7 days open http://bugs.python.org/issue7658 8 Fatal error on thread creation in low memory condition 27 days open http://bugs.python.org/issue7544 8 Direct calls to PyObject_Del/PyObject_DEL are broken for --with 558 days open http://bugs.python.org/issue3299 7 "-3" flag does not work anymore 1 days closed http://bugs.python.org/issue7700 7 tarfile.extractall can't have unicode extraction path 2 days open http://bugs.python.org/issue7693 7 test_urllib: unsetting missing 'env' variable 0 days closed http://bugs.python.org/issue7026 7 os.path.abspath with unicode argument should call os.getcwdu 542 days open http://bugs.python.org/issue3426 From g.brandl at gmx.net Sat Jan 16 21:05:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 16 Jan 2010 21:05:46 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> Message-ID: Am 13.01.2010 21:27, schrieb Lennart Regebro: > On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: >> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >>> What do you need to do in the PYTHONSTARTUP file? >>> Ten years of Python programming, and I didn't even know it existed. :-) >> >> See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. > > Cheese and fries! :-) > > Anyway, I don't see how this could pose any problems, because any user > advanced enough to do these things will be advanced enough to > understand what the problem is and fix it so it works under Python 3 > too. I'd propose adding a bit of text to the environment variables documentation, and especially about the needs of PYTHONSTARTUP files. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From asmodai at in-nomine.org Sat Jan 16 21:50:56 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 16 Jan 2010 21:50:56 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <874ompg225.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> Message-ID: <20100116205056.GL99670@nexus.in-nomine.org> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >hehe. tab completion: With bpython and ipython available, why would you even want to stick to the 'plain old' interactive interpreter? (Sorry to derail the discussion, but maybe there's more people that have not heard of either or both.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B For ever, brother, hail and farewell... From solipsis at pitrou.net Sat Jan 16 21:57:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Jan 2010 20:57:13 +0000 (UTC) Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: Jeroen Ruigrok van der Werven in-nomine.org> writes: > > -On [20100113 22:13], Ralf Schmitt (ralf brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick to the > 'plain old' interactive interpreter? Why wouldn't we? There are probably many more people using the standard Python prompt than ipython. From ben+python at benfinney.id.au Sat Jan 16 23:41:03 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Jan 2010 09:41:03 +1100 Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: <87y6jxk70g.fsf@benfinney.id.au> Jeroen Ruigrok van der Werven writes: > -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick > to the 'plain old' interactive interpreter? Because those optional extras are not always available, whereas the standard interactive interpreter is always available with CPython distributions. -- \ ?I fly Air Bizarre. You buy a combination one-way round-trip | `\ ticket. Leave any Monday, and they bring you back the previous | _o__) Friday. That way you still have the weekend.? ?Steven Wright | Ben Finney From ncoghlan at gmail.com Sun Jan 17 01:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:22:10 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87y6jxk70g.fsf@benfinney.id.au> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> <87y6jxk70g.fsf@benfinney.id.au> Message-ID: <4B525832.8090904@gmail.com> Ben Finney wrote: > Jeroen Ruigrok van der Werven writes: > >> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >>> hehe. tab completion: >> With bpython and ipython available, why would you even want to stick >> to the 'plain old' interactive interpreter? > > Because those optional extras are not always available, whereas the > standard interactive interpreter is always available with CPython > distributions. I've never even contemplated the idea of installing 3rd party apps for the 4 current development builds (2.x trunk, 2.x maintenance, 3.x trunk, 3.x maintenance) on my home development machine. And of course work suffers from the problem of not being allowed to install arbitrary apps I downloaded from the internet without getting the licensing vetted for commercial acceptability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 01:34:08 2010 From: nad at acm.org (Ned Deily) Date: Sat, 16 Jan 2010 16:34:08 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? Message-ID: I've recently seen a couple of references to 3.1.2 go by in checkins which made me wonder whether dates have been proposed yet for updates to either 3.1 or 2.6. I don't recall seeing any and I didn't see any references in the PEPs. Some advance warning would be nice. I assume that some critical problem could trigger planning for an update on short notice; is there a time-limit trigger as well? -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Jan 17 01:55:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:55:38 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52600A.7060201@gmail.com> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Usually there is one more regular maintenance release of the existing maintenance branches within a few months of the creation of the next version (releases before then are at the discretion of the release manager, and security releases continue for much longer). So take the planned 2.7 and 3.2 release dates and add a couple of months to each one to get the likely release dates for the 2.6.x and 3.1.x maintenance releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sun Jan 17 10:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 17 Jan 2010 10:21:49 +0100 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52D6AD.6090302@v.loewis.de> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Barry was once proposing time-based releases; this idea didn't really catch on. It's the release manager who decides when the next bug fix release will be made, often in response to developers asking for one. Regards, Martin From solipsis at pitrou.net Sun Jan 17 14:00:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 13:00:04 +0000 (UTC) Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: Message-ID: Ned Deily acm.org> writes: > > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. There are a couple of release blockers right now. Once they are fixed or deferred, I think it would be nice to have a 3.1.2. Why do you need "some advance warning" though? From ncoghlan at gmail.com Sun Jan 17 14:40:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 23:40:42 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B53135A.7060104@gmail.com> Antoine Pitrou wrote: > Ned Deily acm.org> writes: >> I've recently seen a couple of references to 3.1.2 go by in >> checkins which made me wonder whether dates have been proposed yet >> for updates to either 3.1 or 2.6. I don't recall seeing any and I >> didn't see any references in the PEPs. Some advance warning would >> be nice. > > There are a couple of release blockers right now. Once they are fixed > or deferred, I think it would be nice to have a 3.1.2. Why do you > need "some advance warning" though? Advance warning does allow interested users that would consider upgrading to schedule time for testing before the maintenance release comes out. This is particularly useful in helping to make a 1-week RC period effective in picking up issues that might otherwise lead to a brown paper bag release to fix major issues that slipped through our own automated test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 19:01:37 2010 From: nad at acm.org (Ned Deily) Date: Sun, 17 Jan 2010 10:01:37 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: <4B53135A.7060104@gmail.com> Message-ID: In article <4B53135A.7060104 at gmail.com>, Nick Coghlan wrote: > Antoine Pitrou wrote: > > Ned Deily acm.org> writes: > >> I've recently seen a couple of references to 3.1.2 go by in > >> checkins which made me wonder whether dates have been proposed yet > >> for updates to either 3.1 or 2.6. I don't recall seeing any and I > >> didn't see any references in the PEPs. Some advance warning would > >> be nice. > > There are a couple of release blockers right now. Once they are fixed > > or deferred, I think it would be nice to have a 3.1.2. Why do you > > need "some advance warning" though? > > Advance warning does allow interested users that would consider > upgrading to schedule time for testing before the maintenance release > comes out. This is particularly useful in helping to make a 1-week RC > period effective in picking up issues that might otherwise lead to a > brown paper bag release to fix major issues that slipped through our own > automated test coverage. That. and resource contention: there are always potential fixes in the pipeline that could or should be bumped in priority if one knows there is a code cutoff approaching. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Sun Jan 17 20:51:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 20:51:58 +0100 Subject: [Python-Dev] Enhancing the shutil module Message-ID: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Hello, For 2.7/3.2, I am in the process of removing modules in Distutils that can be replaced by calls to existing functions in stdlib. For instance, "dir_util" and "file_util" (old modules from the Python 1.x era) are going away in favor of calls to shutil (and os), so the Distutils package gets lighter. Another module I would like to move away from Distutils is "archive_util". It contains helpers to build archives, whether they are zip or tar files. I propose to move those useful functions into shutil, as this seems the most logical place. I also propose to maintain this "shutil" module for now on (no one is declared as a maintainer in maintainers.rst) since Distutils will become a heavy user of its functions. Any objections/opinions ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 20:53:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 19:53:03 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536A9F.5050203@voidspace.org.uk> On 17/01/2010 19:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > > Regards, > Tarek > > I think it's a great idea. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Sun Jan 17 20:55:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2010 11:55:47 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > If it's archive-agnostic then shutil is probably the best place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Great! > Any objections/opinions ? > No objections from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jan 17 21:04:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 20:04:41 +0000 (UTC) Subject: [Python-Dev] Enhancing the shutil module References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: Tarek Ziad? gmail.com> writes: > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. Are these functions portable? Do they rely on external programs? > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. It's ok for me. Regards Antoine. From ziade.tarek at gmail.com Sun Jan 17 21:09:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:09:18 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: > > > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >> >> Hello, >> >> For 2.7/3.2, I am in the process of removing modules in Distutils that >> can be replaced by calls to existing functions in stdlib. For >> instance, "dir_util" and "file_util" (old modules from the Python 1.x >> era) are going away in favor of calls to shutil (and os), so the >> Distutils package gets lighter. >> >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. > > If it's archive-agnostic then shutil is probably the best place. In more details: It allows the creation of gzip, bzip2, tar and zip files through a single API. There's a registry of supported formats and the API is driven by a format identifier. To do the work it uses stdlib's compression modules. Although it tries the "zip" system command as a fallback if the "zipfile" module is not present. (notice that I've removed the support of "compress" (.Z) some time ago) Regards Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 21:15:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 20:15:00 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536FC4.9090304@voidspace.org.uk> On 17/01/2010 20:04, Antoine Pitrou wrote: > Tarek Ziad? gmail.com> writes: > >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. >> > Are these functions portable? Do they rely on external programs? > > I believe that part of the work that Tarek has been doing has been to make these distutils commands use the Python standard library and not depend on external programs. In which case they seem like *excellent* additions to the shutil module. Of course Tarek can speak for himself... Michael >> I also propose to maintain this "shutil" module for now on (no one is >> declared as a maintainer in maintainers.rst) since Distutils will >> become a heavy user of its functions. >> > It's ok for me. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Sun Jan 17 21:43:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:43:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B536FC4.9090304@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> Message-ID: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> On Sun, Jan 17, 2010 at 9:15 PM, Michael Foord wrote: [..] >> Are these functions portable? Do they rely on external programs? >> >> > > I believe that part of the work that Tarek has been doing has been to make > these distutils commands use the Python standard library and not depend on > external programs. In which case they seem like *excellent* additions to the > shutil module. yes, in the past the "tar" files where created using the "tar" command but this has been changed. For a while now, they are portable and they use stdlib code only. A recent addition is to be able to define user/group permissions in the tar files, thanks to Lars' work in the tarfile module. There's one remaining external call for "zip" done if the zip module is not found, but I am happy to remove it and throw an exception if it's not found, and keep the external "zip" call on Distutils side, so shutil stays 100% stdlib-powered. > Of course Tarek can speak for himself... Thanks for explaining it ! :) Regards Tarek -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Sun Jan 17 22:50:52 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sun, 17 Jan 2010 13:50:52 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> Message-ID: <4B53863C.5060304@activestate.com> On 1/17/2010 12:09 PM, Tarek Ziad? wrote: > On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: >> > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >>> >> Another module I would like to move away from Distutils is >>> >> "archive_util". It contains helpers to build archives, whether they >>> >> are zip or tar files. I propose to move those useful functions into >>> >> shutil, as this seems the most logical place. >> > If it's archive-agnostic then shutil is probably the best place. > In more details: > It allows the creation of gzip, bzip2, tar and zip files through a single API. > There's a registry of supported formats and the API is driven by a > format identifier. Will it also allow decompression of the said archive types? Distribute has some utility code to handle zip/tar archives. So does PyPM. This is because the `tarfile` and `zipfile` modules do not "just work" due to several issues. See http://gist.github.com/279606 Take note of the following in the above code: 1) _ensure_read_write_access 2) *File.is_valid 3) ZippedFile.extract ... issue 6510 4) ZippedFile.extract ... issue 6609 5) TarredFile.extract ... issue 6584 6) The way unpack() detects the unpacked directory. -srid From ziade.tarek at gmail.com Sun Jan 17 23:09:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 23:09:29 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B53863C.5060304@activestate.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> <4B53863C.5060304@activestate.com> Message-ID: <94bdd2611001171409j4ec1e0bfj47e59894fdec0d8a@mail.gmail.com> On Sun, Jan 17, 2010 at 10:50 PM, Sridhar Ratnakumar wrote: [..] > Will it also allow decompression of the said archive types? No but it would make sense having this one as well. Distribute/Setuptools' "unpack_archive" (used by easy_install) was implemented using the same principle as Distutils' "make_archive". I can add it as well while I am at it : it has been successfully used for years by easy_install. > Distribute has > some utility code to handle zip/tar archives. So does PyPM. This is because > the `tarfile` and `zipfile` modules do not "just work" due to several > issues. > > See http://gist.github.com/279606 > > Take note of the following in the above code: > > ?1) _ensure_read_write_access > ?2) *File.is_valid > ?3) ZippedFile.extract ... issue 6510 > ?4) ZippedFile.extract ... issue 6609 > ?5) TarredFile.extract ... issue 6584 Looks like some of these are already fixed (I just looked quickly in the bug tracker though) If its not already done and pending, it would be great if you could refactor your fixes into patches for the remaining issues for each one of those modules Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Sun Jan 17 23:56:37 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 17 Jan 2010 17:56:37 -0500 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: <4B52D6AD.6090302@v.loewis.de> References: <4B52D6AD.6090302@v.loewis.de> Message-ID: On Jan 17, 2010, at 4:21 AM, Martin v. L?wis wrote: > Barry was once proposing time-based releases; this idea didn't really > catch on. I'm still a proponent of this, but it doesn't seem like there's enough support for it. > It's the release manager who decides when the next bug fix release will > be made, often in response to developers asking for one. I'm happy to make a 2.6.5 release whenever it makes sense. -Barry From ncoghlan at gmail.com Mon Jan 18 13:40:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Jan 2010 22:40:01 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> Message-ID: <4B5456A1.2080109@gmail.com> Tarek Ziad? wrote: > There's one remaining external call for "zip" done if the zip module > is not found, but I am happy to remove it and throw an exception if > it's not found, and keep the external "zip" call on Distutils side, so > shutil stays 100% stdlib-powered. +1 for that approach. These changes all sound like nice additions to shutil, and hooray for every module that gets adopted by an active maintainer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From masklinn at masklinn.net Mon Jan 18 14:34:14 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 14:34:14 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5456A1.2080109@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> Message-ID: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: > > Tarek Ziad? wrote: >> There's one remaining external call for "zip" done if the zip module >> is not found, but I am happy to remove it and throw an exception if >> it's not found, and keep the external "zip" call on Distutils side, so >> shutil stays 100% stdlib-powered. > > +1 for that approach. These changes all sound like nice additions to > shutil, and hooray for every module that gets adopted by an active > maintainer :) Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. From doug.hellmann at gmail.com Mon Jan 18 14:46:05 2010 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 18 Jan 2010 08:46:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils >>> side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil > advertises itself as "a number of high-level operations on files and > collections of files." and from what I understood it was a bunch of > shell-type utility functions to easily copy, move or remove files > and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/ > objects in a new module separate from shutil, dealing specifically > with cross-archive APIs and linked from the current archive-specific > modules (essentially, just take the current archive_util, move it to > the toplevel of the stdlib and maybe rename it)? It would also make > the module much easier to find when searching through the module > listing, I think. +1 From fuzzyman at voidspace.org.uk Mon Jan 18 14:57:49 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 18 Jan 2010 13:57:49 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> Message-ID: <4B5468DD.5040200@voidspace.org.uk> On 18/01/2010 13:46, Doug Hellmann wrote: > > On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > >> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>> >>> Tarek Ziad? wrote: >>>> There's one remaining external call for "zip" done if the zip module >>>> is not found, but I am happy to remove it and throw an exception if >>>> it's not found, and keep the external "zip" call on Distutils side, so >>>> shutil stays 100% stdlib-powered. >>> >>> +1 for that approach. These changes all sound like nice additions to >>> shutil, and hooray for every module that gets adopted by an active >>> maintainer :) >> >> Isn't it a bit weird to include that to shutil though? shutil >> advertises itself as "a number of high-level operations on files and >> collections of files." Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> and from what I understood it was a bunch of shell-type utility functions like tar and zip? :-) >> to easily copy, move or remove files and directories (that's pretty >> much all there is in it at this time). >> I don't think the additions are out of place prima-facie. >> Wouldn't it make more sense to put those "archive utils" >> functions/objects in a new module separate from shutil, dealing >> specifically with cross-archive APIs and linked from the current >> archive-specific modules (essentially, just take the current >> archive_util, move it to the toplevel of the stdlib and maybe rename >> it)? It would also make the module much easier to find when searching >> through the module listing, I think. > > +1 > Proliferation of modules is itself a bad thing though. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Mon Jan 18 15:34:12 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 15:34:12 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <94bdd2611001180634sacd998fm6f67dc680e2e61c3@mail.gmail.com> On Mon, Jan 18, 2010 at 2:57 PM, Michael Foord wrote: [..] >>> Wouldn't it make more sense to put those "archive utils" >>> functions/objects in a new module separate from shutil, dealing specifically >>> with cross-archive APIs and linked from the current archive-specific modules >>> (essentially, just take the current archive_util, move it to the toplevel of >>> the stdlib and maybe rename it)? It would also make the module much easier >>> to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. I am with Michael here. I think having this function in shutil is the right place. For the find problem, I think docs.python.org is easy to search now, as long as the shutil module documentation has an good set of examples for this new API. We could even add references in the tarfile, zipfile modules documentation pointing to these examples. Regards Tarek -- Tarek Ziad? | http://ziade.org From masklinn at masklinn.net Mon Jan 18 15:56:05 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 15:56:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> On 18 Jan 2010, at 14:57 , Michael Foord wrote: > > On 18/01/2010 13:46, Doug Hellmann wrote: >> >> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: >> >>> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>>> >>>> Tarek Ziad? wrote: >>>>> There's one remaining external call for "zip" done if the zip module >>>>> is not found, but I am happy to remove it and throw an exception if >>>>> it's not found, and keep the external "zip" call on Distutils side, so >>>>> shutil stays 100% stdlib-powered. >>>> >>>> +1 for that approach. These changes all sound like nice additions to >>>> shutil, and hooray for every module that gets adopted by an active >>>> maintainer :) >>> >>> Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." > > Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? > Well no those are high-level operations on a very restricted set of file types (archives). >>> and from what I understood it was a bunch of shell-type utility functions > > like tar and zip? :-) > Tar and zip have a module each at this time, so they're considered pretty big. I don't think anybody would consider "mv" big enough to give it a module on its own. >>> Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. From phd at phd.pp.ru Mon Jan 18 16:03:38 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 18 Jan 2010 18:03:38 +0300 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118150337.GA19391@phd.pp.ru> On Mon, Jan 18, 2010 at 02:34:14PM +0100, Masklinn wrote: > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. +1 Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ziade.tarek at gmail.com Mon Jan 18 16:24:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 16:24:37 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> Message-ID: <94bdd2611001180724u7f48e620ub32e13ef96c873b9@mail.gmail.com> On Mon, Jan 18, 2010 at 3:56 PM, Masklinn wrote: [..] >> Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> > Well no those are high-level operations on a very restricted set of file types (archives) not really: make_archive/unpack_archive, are also dealing with files and directories. [..] >> Proliferation of modules is itself a bad thing though. > Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. > > Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. I am not sure why this would happen. For instance, shutil is already on the top of the os module since a few major versions IIRC, because it reads and writes files and directories. But it was not moved into the os package (or vice-versa) The shutil module uses APIs to read and write files. So if it works with archives, it's just a specific read/write API that is used, but that doesn't mean tarfile and zipfile might be reunited with shutil imho. If the shutil module is restricted to high-level files and directories manipulation, working with archives is just a target like another I think. But at the end I am 0- to create a new module, because what really matters to me is to take it out of Distutils :) Regards, Tarek -- Tarek Ziad? | http://ziade.org From listsin at integrateddevcorp.com Mon Jan 18 16:56:05 2010 From: listsin at integrateddevcorp.com (Steve Steiner (listsin)) Date: Mon, 18 Jan 2010 10:56:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. As much of a pain as it is to get new modules accepted, I agree that mixing archiving functions into shutil is not the right way to do it and that a separate archive_util module would make much more sense and would give a logical place to put any extensions to archive handling. S From rdmurray at bitdance.com Mon Jan 18 20:28:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 18 Jan 2010 14:28:47 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: > As much of a pain as it is to get new modules accepted, I agree that > mixing archiving functions into shutil is not the right way to do it > and that a separate archive_util module would make much more sense and > would give a logical place to put any extensions to archive handling. Looking at the source code and API for both shutil and archive_util, I think that the archive_util methods fit into shutil. shutil currently wraps some standard library facilities with convenience functions for operations you might otherwise perform at the shell command line using OS facilities. As far as I can tell, archive_util does the same, and seems quite within the shutil mission of "high level file operations". So +1 from me for putting these in shutil. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From p.f.moore at gmail.com Mon Jan 18 20:48:43 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 18 Jan 2010 19:48:43 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> Message-ID: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> 2010/1/18 R. David Murray : > On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: >> As much of a pain as it is to get new modules accepted, I agree that >> mixing archiving functions into shutil is not the right way to do it >> and that a separate archive_util module would make much more sense and >> would give a logical place to put any extensions to archive handling. > > Looking at the source code and API for both shutil and archive_util, I > think that the archive_util methods fit into shutil. ?shutil currently > wraps some standard library facilities with convenience functions > for operations you might otherwise perform at the shell command line using > OS facilities. ?As far as I can tell, archive_util does the same, and > seems quite within the shutil mission of "high level file operations". > > So +1 from me for putting these in shutil. Conceptually, I'm happy with these going into shutil (and +1 on the rest of Tarek's proposal, too!) To my mind, shutil is a module for higher-level operations on files - the sort of things you'd do in shell commands, like move a batch of files around (mv), create a directory tree (mkdir -p). Tarring or zipping up a batch of files fits nicely into that space. Paul. From david.lyon at preisshare.net Tue Jan 19 02:53:43 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 18 Jan 2010 20:53:43 -0500 Subject: [Python-Dev] PyCon Keynote In-Reply-To: References: Message-ID: <0dfb370163cf3a284ca256f49662db74@preisshare.net> On Wed, 13 Jan 2010 10:51:14 -0800, Guido van Rossum wrote: > Please mail me topics you'd like to hear me talk about in my keynote > at PyCon this year. As a typical (but perphaps more vocal) python developer I would like to hear things that might aspire me and others to move to python 3. The way I see it is that python 2.x represents everyones comfort zone. It's safe and nobody got fired at work for using python 2.x I would like to hear how python 3 could be 'shaken' up with a slightly less conservative policy that has gone with the python 2.x series. The logic perhaps being that since only a minority use the 3.x series, it's functionality set is still more up in the air. imo it needs bigger batteries to give it more power than the 2.x series. This meaning that the stdlib could take an extra 5-10 packages not in the 2.x series. Just as one example, sphinx - a great documentation tool. I can easily name five or six others. I'd also like to hear more of your ideas on pypi and getting it to have as much who-ha as CPAN. You could do a lot to enlarge the developer group. Help us all get our priorities to be inline with your own wishes and expectations. If you ask us all to put in a big year and buy you a beer at the end.. I'm certain we all will. Every little bit of inspiration and direction counts for a lot... Best Regards David From ncoghlan at gmail.com Tue Jan 19 12:20:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Jan 2010 21:20:05 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> Message-ID: <4B559565.8090602@gmail.com> Paul Moore wrote: > 2010/1/18 R. David Murray : >> So +1 from me for putting these in shutil. > > Conceptually, I'm happy with these going into shutil (and +1 on the > rest of Tarek's proposal, too!) > > To my mind, shutil is a module for higher-level operations on files - > the sort of things you'd do in shell commands, like move a batch of > files around (mv), create a directory tree (mkdir -p). Tarring or > zipping up a batch of files fits nicely into that space. This is also reflected in the way at least Windows handles archives these days - it took them a couple of iterations to get it right (and resolve some of the performance impacts), but Explorer now does a decent job of integrating archives into the directory tree as "folders that happen to be compressed". Are archives as fundamental as directories and files? No. But in the context of shutil, the fact that their internal structure is largely about directories and files makes them more than just another arbitrary file type. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 19 14:16:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 08:16:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad Message-ID: <20100119081639.5d431ed9@freewill> I've just updated the Launchpad mirrors for the 4 active Python branches, trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar branches on code.python.org but it's probably been 7 months or so since those were regularly updated. Now the Launchpad branches sync against the read-only Subversion branches at http://svn.python.org, so they should remain up-to-date (within the re-sync timeframe of about 4 hours). This means you can once again use Bazaar to get local branches of Python, and you can of course push your own branches to Launchpad. I believe you can even use the bzr-svn plugin to commit changes back to the Subversion master, though I have not yet tried this. To get a local branch, just do any of the following: % bzr branch lp:python (for trunk) % bzr branch lp:python/2.6 % bzr branch lp:python/py3k % bzr branch lp:python/3.1 (It's fairly easy to create new mirrors for other Subversion branches, e.g. Python 2.5; just drop me an email if you want them.) If you're going to create a lot of branches you probably want to put them in a shared repository. E.g. % bzr init-repo pythonbzr % cd pythonbzr % bzr branch lp:python/py3k Bazaar 2.0 or better is recommended. For me, it took about 5m to check the first branch out from Launchpad, and then about 30s or so for each subsequent branch. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From abpillai at gmail.com Tue Jan 19 14:37:18 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Tue, 19 Jan 2010 19:07:18 +0530 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <8548c5f31001190537l2bcab5cfkb6f461910e4abd8b@mail.gmail.com> On Mon, Jan 18, 2010 at 1:21 AM, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > +1 for this. Just make sure that you change the docstring of shutil which now reads as, " shutil - Utility functions for copying files and directory trees." According to this "definition", archives don't fit in there. But the functionality does fit right in, so just need to make sure that it is reflected in the __doc__ . > Regards, > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jan 19 16:50:42 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Jan 2010 15:50:42 +0000 (UTC) Subject: [Python-Dev] Mailing List archive corruption? Message-ID: Hi, When I look at the mailing list archive for python-dev, I see some odd stuff at the bottom of the page: http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 Anyone know what's happened? Regards, Vinay Sajip From barry at python.org Tue Jan 19 17:07:48 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:07:48 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: Message-ID: <20100119110748.69bc564a@freewill> On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: >When I look at the mailing list archive for python-dev, I see some odd stuff at >the bottom of the page: > >http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >Anyone know what's happened? WTF? I think the archives were recently regenerated, so there's probably a fubar there. CC'ing the postmasters. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From foom at fuhm.net Tue Jan 19 17:24:57 2010 From: foom at fuhm.net (James Y Knight) Date: Tue, 19 Jan 2010 11:24:57 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119110748.69bc564a@freewill> References: <20100119110748.69bc564a@freewill> Message-ID: On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > >> When I look at the mailing list archive for python-dev, I see some >> odd stuff at >> the bottom of the page: >> >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 >> >> Anyone know what's happened? > > WTF? I think the archives were recently regenerated, so there's > probably a > fubar there. CC'ing the postmasters. That happens if messages had unescaped "From" lines in the middle of them. No doubt, you've now broken every link anyone had ever made into the python-dev archives, because now all the article numbers are different. BTDT...unfortunately... Pipermail really is quite crappy, sigh. Anyhow, when I did that, I went back to a backup to get the original article numbers, and edited the mbox file escaping From lines or adding additional empty messages until the newly regenerated article numbers matched the originals. I'd highly recommend going through that painful process, since I suspect a *lot* of people have links to the python-dev archive. Hope you have a backup (or can find caches on google or archive.org or something). James From barry at python.org Tue Jan 19 17:44:21 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:44:21 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119114421.3b96fbd4@freewill> On Jan 19, 2010, at 11:24 AM, James Y Knight wrote: >No doubt, you've now broken every link anyone had ever made into the >python-dev archives, because now all the article numbers are >different. BTDT...unfortunately... Pipermail really is quite crappy, >sigh. I've been trying for 10+ years to get folks interested in helping me fix this (and a few other warts) in Pipermail, to not much success. ;/ >Anyhow, when I did that, I went back to a backup to get the original >article numbers, and edited the mbox file escaping From lines or >adding additional empty messages until the newly regenerated article >numbers matched the originals. I'd highly recommend going through that >painful process, since I suspect a *lot* of people have links to the >python-dev archive. Hope you have a backup (or can find caches on >google or archive.org or something). bin/cleanarch uses a set of heuristics to find unescaped From lines and fix them. It's generally pretty good, but it certain can change message numbers (and sadly, their urls). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rdmurray at bitdance.com Tue Jan 19 18:48:41 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 19 Jan 2010 12:48:41 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119174841.9BC3B16A53@kimball.webabinitio.net> On Tue, 19 Jan 2010 11:24:57 -0500, James Y Knight wrote: > > On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > > > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > > > >> When I look at the mailing list archive for python-dev, I see some > >> odd stuff at the bottom of the page: > >> > >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >> > >> Anyone know what's happened? > > > > WTF? I think the archives were recently regenerated, so there's > > probably a fubar there. CC'ing the postmasters. > > That happens if messages had unescaped "From" lines in the middle of > them. > > No doubt, you've now broken every link anyone had ever made into the > python-dev archives, because now all the article numbers are > different. BTDT...unfortunately... Pipermail really is quite crappy, > sigh. > > Anyhow, when I did that, I went back to a backup to get the original > article numbers, and edited the mbox file escaping From lines or > adding additional empty messages until the newly regenerated article > numbers matched the originals. I'd highly recommend going through that > painful process, since I suspect a *lot* of people have links to the > python-dev archive. Hope you have a backup (or can find caches on > google or archive.org or something). The Python issue tracker does, for one. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From ncoghlan at gmail.com Tue Jan 19 22:18:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 07:18:52 +1000 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119174841.9BC3B16A53@kimball.webabinitio.net> References: <20100119110748.69bc564a@freewill> <20100119174841.9BC3B16A53@kimball.webabinitio.net> Message-ID: <4B5621BC.4070608@gmail.com> R. David Murray wrote: > The Python issue tracker does, for one. And all the PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at pythontest.org Wed Jan 20 00:16:44 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 10:16:44 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119081639.5d431ed9@freewill> References: <20100119081639.5d431ed9@freewill> Message-ID: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Hi Barry, That looks very interesting... So does that mean we could update the stdlib for a given python version using this ? David > I've just updated the Launchpad mirrors for the 4 active Python branches, > trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar > branches > on code.python.org but it's probably been 7 months or so since those were > regularly updated. Now the Launchpad branches sync against the read-only > Subversion branches at http://svn.python.org, so they should remain > up-to-date > (within the re-sync timeframe of about 4 hours). > > This means you can once again use Bazaar to get local branches of Python, > and > you can of course push your own branches to Launchpad. I believe you can > even > use the bzr-svn plugin to commit changes back to the Subversion master, > though > I have not yet tried this. > > To get a local branch, just do any of the following: > > % bzr branch lp:python (for trunk) > % bzr branch lp:python/2.6 > % bzr branch lp:python/py3k > % bzr branch lp:python/3.1 > > (It's fairly easy to create new mirrors for other Subversion branches, > e.g. Python 2.5; just drop me an email if you want them.) > > If you're going to create a lot of branches you probably want to put them > in a > shared repository. E.g. > > % bzr init-repo pythonbzr > % cd pythonbzr > % bzr branch lp:python/py3k > > Bazaar 2.0 or better is recommended. For me, it took about 5m to check > the > first branch out from Launchpad, and then about 30s or so for each > subsequent > branch. > > Enjoy, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/david.lyon%40pythontest.org > From barry at python.org Wed Jan 20 00:54:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 18:54:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119185439.299660c7@freewill> On Jan 20, 2010, at 10:16 AM, David Lyon wrote: >Hi Barry, > >That looks very interesting... Hi David, >So does that mean we could update the stdlib for a given >python version using this ? In a sense, yes (if I understand your question correctly). You can use Bazaar to branch any of the 4 Python series and use all the modern DVCS goodness to develop your updates. You can share your branches with others via e.g. Launchpad and even request reviews (called "merge proposals") to get feedback from others. The one thing I am unsure about, mostly because I have not tried it, is whether your Bazaar branch can be used to commit directly back to the Python Subversion master branches. I /think/ the answer is yes, assuming of course that you have permission to do so, and that you have a modern version of Bazaar and the bzr-svn plugin. It might even be possible to commit your Bazaar branch to a local Subversion branch, and then commit the latter to get it pushed up to svn.python.org. At worst, you would use Bazaar's features to get your patch into a state you and your reviewers are happy with, then you would generate a diff for application to your copy of the svn.python.org branch. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 01:51:23 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 11:51:23 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119185439.299660c7@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> Message-ID: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> > On Jan 20, 2010, at 10:16 AM, Barry wrote: >> So does that mean we could update the stdlib for a given >> python version using this ? > > In a sense, yes (if I understand your question correctly). Yeah, it just needs an implementation. > The one thing I am unsure about, mostly because I have not tried it, is > whether your Bazaar branch can be used to commit directly back to the > Python Subversion master branches. I /think/ the answer is yes, > assuming of course that you have permission to do so... Well I'm too Senior and my stuff is too forward looking to qualify for that just yet. I'd be happy to see bzr and mercurial and git all made it together into the stdlib for python 3. That would give a superb updating mechanism for python that would propel python well beyond the dinosaur badlands of CPAN and other languages. I was actually reading from (http://en.wikipedia.org/wiki/Python_%28programming_language%29): "Rather than requiring all desired functionality to be built into the language's core, Python was designed to be highly extensible. .. .. This design of a small core language with a large standard library and an easily extensible interpreter was intended by Van Rossum from the very start because of his frustrations with ABC (which espoused the opposite mindset).[5]" To me, the source code control systems seem to be fully in tune with the original design of python. That is, to be able to easily pull external libraries in. I think what has changed is that the mechanisms now (the SCMs) are way more highly developed than before. Apart from that though, after reading the full wikipedia article I'm left with the distinct impression that things are still pretty much the same (in that python design philosophy is advanced), just that the landscape (of external C libraries) has changed. Now all the libraries are external (on the internet) and all externally managed. So with just a tiny amount of work, imho we could pull it all together to bring python 3 *back* to being that cool tool that it once was (not saying it isn't now). Were you offering me an experimental branch somewhere for python 3 SCM integration ? David From jnoller at gmail.com Wed Jan 20 02:09:15 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 20:09:15 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> On Tue, Jan 19, 2010 at 7:51 PM, David Lyon wrote: >> On Jan 20, 2010, at 10:16 AM, Barry wrote: > >>> So does that mean we could update the stdlib for a given >>> python version using this ? >> >> In a sense, yes (if I understand your question correctly). > > Yeah, it just needs an implementation. > >> The one thing I am unsure about, mostly because I have not tried it, is >> whether your Bazaar branch can be used to commit directly back to the >> Python Subversion master branches. ?I /think/ the answer is yes, >> assuming of course that you have permission to do so... > > Well I'm too Senior and my stuff is too forward looking to qualify > for that just yet. > > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. i sincerely doubt that a source control system will be included in the standard library in the future. Especially 3. A SCM is not a "package management system". Barry was talking about mirrors of the python code. It is true a "package manager" could be developed based on a SCM, however you need to implement this far away from the stdlib and get traction with it within the community long before inclusion would be considered. The decision to move python's source control from SVN to mercurial was controversial enough; including 3 or more scm libraries into core would be an intractable uphill mountain of bike sheds. > So with just a tiny amount of work, imho we could pull > it all together to bring python 3 *back* to being that > cool tool that it once was (not saying it isn't now). Python 3 is still modularized, still has a standard library, etc. If you're really interested in helping with the standard library, get on stdlib-sig, and get ready to write code and PEPs. > Were you offering me an experimental branch somewhere > for python 3 SCM integration ? Barry made bzr mirrors of the python svn tree. Not a python with bzr included. jesse From barry at python.org Wed Jan 20 04:32:30 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 22:32:30 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <20100119223230.4c4a7ed5@freewill> On Jan 19, 2010, at 08:09 PM, Jesse Noller wrote: >The decision to move python's source control from SVN to mercurial was >controversial enough; including 3 or more scm libraries into core >would be an intractable uphill mountain of bike sheds. I'd be surprised if any of the big 3 DVCS developers would actually /want/ their stuff in the stdlib. Being in the stdlib has its advantages and disadvantages. I think for rapidly developing technology, the latter can actually outweigh the former. (Besides, git in the stdlib doesn't make much sense :). >Barry made bzr mirrors of the python svn tree. Not a python with bzr included. Bingo. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 04:43:12 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 14:43:12 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> > On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: > Python 3 is still modularized, still has a standard library, etc. If > you're really interested in helping with the standard library, get on > stdlib-sig, and get ready to write code and PEPs. Thank you for your direction to move these items forward to PEPs and Code. > i sincerely doubt that a source control system will be included in the > standard library in the future. Especially 3. Yeah and who twenty years ago thought you would get a 1GB memory card for $3 when all we had was 10Meg hard disks and they were the full 8" platter. > A SCM is not a "package management system". Exactly. It almost makes the need for a "package management system" pretty much obsolete if you can update your code directly from the developers sources. That's what all these SCMs provide. Plus it's addictive. It's hard to go back to 'package' style technology once you have all your code on an SCM based feed. > Barry was talking about mirrors of the python code. It is true a > "package manager" could be developed based on a SCM, however you need > to implement this far away from the stdlib and get traction with it > within the community long before inclusion would be considered. I think I'll have better chances with PEPs. Being honest, if wonderful libraries like Sphinx and Mercurial and Git and BZR can't make it into the stdlib, then there is no hope for even newer code to get in there. Plus, promoting all sorts of new and fangled tools however good they may or may not be just confuses users and ends up being a waste of time imho. It isn't good management of volounteers time and effort. If you could imagine disaster relief coordinated this way, it would just be a disaster in itself. That's why it has taken some 5 years to get PEP-345 done. > The decision to move python's source control from SVN to mercurial was > controversial enough; including 3 or more scm libraries into core > would be an intractable uphill mountain of bike sheds. Not at all. It would be a very fair thing to do. Not to mention being great for users. > Barry made bzr mirrors of the python svn tree. Not a python with bzr > included. I can't resist asking for that again.. I heard it only in Monty speek. Did you just say ?: "Barry made a bizarre mirror of the python suvern tree. Not a python with a buzzer included." Anyway.. Maybe I do get what your talking about. Even if you do talk with a strange accent. :-) David From jnoller at gmail.com Wed Jan 20 05:10:07 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 23:10:07 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001192010v66b728dbxa7ad1ed1fc1df15f@mail.gmail.com> On Tue, Jan 19, 2010 at 10:43 PM, David Lyon wrote: [snip] > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. Did you ever stop to think that some package authors do not want their code in the standard library? That throwing random shiny things in there just makes it a junk drawer? Besides: Show us the PEP to include Sphinx, and it's dependencies in the standard lib, with Georg's signature at the bottom. The authors of modules have to want things to be in there, they have to be best of breed, tested or common-enough patterns to warrant a slot. We have things in the standard lib which still need more TLC and maintenance, or they need to be shunted into space. Adding random things, which may or may not help packaging, and installing things from random places in someone source repository won't make things better. > Plus, promoting all sorts of new and fangled tools however > good they may or may not be just confuses users and ends > up being a waste of time imho. It isn't good management > of volounteers time and effort. > > If you could imagine disaster relief coordinated this > way, it would just be a disaster in itself. > > That's why it has taken some 5 years to get PEP-345 done. I'm going to assume that you're trolling now, or intentionally misrepresenting facts. Maybe a little of both. A PEP, and an implementation and the ability to rationally debate, discuss and defend your proposal is what is needed to enact changes on policies, python-core or the standard library. >> The decision to move python's source control from SVN to mercurial was >> controversial enough; including 3 or more scm libraries into core >> would be an intractable uphill mountain of bike sheds. > > Not at all. > > It would be a very fair thing to do. Not to mention being > great for users. There should be one-- and preferably only one --obvious way to do it. > Anyway.. Maybe I do get what your talking about. Even if you do > talk with a strange accent. :-) My sense of humor has been disabled by repeated stunning at your hands. I admire your enthusiasm, even if I do think some of it is misplaced, or at guided into the proper channels at very least. Please, you seem to have the time and willingness to help, please go about this the right way. Discuss things on the proper lists, make concrete proposals. If you have have standard lib changes, discuss them on stdlib-sig, if you have ideas about python-the-language, or the interpreter, etc - please discuss it on python-ideas. jesse From ben+python at benfinney.id.au Wed Jan 20 05:16:25 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 15:16:25 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <87wrzdfm1y.fsf@benfinney.id.au> "David Lyon" writes: > Being honest, if wonderful libraries like Sphinx and Mercurial and Git > and BZR can't make it into the stdlib, then there is no hope for even > newer code to get in there. Those are applications, not libraries. Applications don't belong in the standard library. -- \ ?If you pick up a starving dog and make him prosperous, he will | `\ not bite you. This is the principal difference between a dog | _o__) and a man.? ?Mark Twain, _Pudd'n'head Wilson_ | Ben Finney From brian.curtin at gmail.com Wed Jan 20 05:19:47 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 19 Jan 2010 22:19:47 -0600 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: On Tue, Jan 19, 2010 at 21:43, David Lyon wrote: > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > I'm not entirely sure I see why the inclusion of a SCM into the stdlib is necessary. Just because pieces of software are mature and proven in their fields doesn't mean we should add them, or that them *not* being in the stdlib should be a basis for other projects making it in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Jan 20 05:26:44 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:26:44 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119232644.7fa25691@freewill> On Jan 20, 2010, at 02:43 PM, David Lyon wrote: >> On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: >> A SCM is not a "package management system". > >Exactly. It almost makes the need for a "package management system" >pretty much obsolete if you can update your code directly from >the developers sources. > >That's what all these SCMs provide. Plus it's addictive. It's >hard to go back to 'package' style technology once you have >all your code on an SCM based feed. Well... I'm not so sure. A package management system like apt does a /ton/ of additional bookkeeping and work to ensure a robust, highly consistent, functioning system. And while both Python and most Linux distributions have their own notion of "package management", they don't always play nicely together. Tarek and the distutils-sig's work is trying to make the world a better place by bridging this gap better, and there is code out there that makes it easier to say import a Python package from the Cheeseshop and .deb-ify it for use on Debian and Ubuntu. There's also work being done in Launchpad that will allow you to "build-from-branch" so that in a sense you could let a build farm take your Bazaar branches and automatically build the packages from them. I've strayed off-topic I suppose, but I see SCMs and package managers as complementary technologies that help with important parts of the process of delivering software to end-users, but I don't quite see how one can make the other obsolete. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 05:29:34 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 15:29:34 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> > On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: > > I'd be surprised if any of the big 3 DVCS developers would actually /want/ > their stuff in the stdlib. If they ask, they'll get told they're motorbike-shedding. "It's better if their users ask". So here I am as a user doing things the 'right' way. > Being in the stdlib has its advantages and > disadvantages. I think for rapidly developing technology, the > latter can actually outweigh the former. If it's about being able to do updates, then I think this resolves an old and circular argument. As the SCM implementation would, one would expect, to be able to update itself. Side benefits are that it can update everything else along with it at the same time. User Apps, Packages, whatever. It's even better having SCM in an Industrial/Scientific environment. Here's an example: - a machine breaks.. (I mean the software for/in it) - you fix the code, maybe on the spot - you commit and push back to the repository - your code gets checked in and run through the testbot and then you get blamed and have to do the whole thing again properly with a test case. Oh well.. Well anyway, whatever you guys might say, that's a whole lot more efficient than running back to the development machine and going through some obscure build and test and publish process to do a fix on a production machine. Point : The fact that SCMs are two way is great in a production environment. No packaging solution can come close. So why not have python SCMs included as batteries in python.. All these arguments I can take off to the stdlib list when I get the chance.. David From barry at python.org Wed Jan 20 05:50:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:50:36 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119235036.57f6e867@freewill> Okay, last follow up on this and then I'm going to bed. :) On Jan 20, 2010, at 03:29 PM, David Lyon wrote: >> On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: >> >> I'd be surprised if any of the big 3 DVCS developers would actually /want/ >> their stuff in the stdlib. > >If they ask, they'll get told they're motorbike-shedding. "It's better >if their users ask". So here I am as a user doing things the 'right' >way. Actually, you're not. It's not up to the Python community to initiate this. If you really want this, you should engage with the relevant DVCS communities and push them to request it. >Side benefits are that it can update everything else along >with it at the same time. User Apps, Packages, whatever. I get that. Heck, I still run one Gentoo server which I think is as close to the edge you're describing as I'm comfortable with. It's all great until the wheels come off and then it can take *days* to get a functioning system again. The big difference is that I rely on my DVCS to keep one small thing, or a few variants of the same thing, all sane. But I rely on my distribution vendor to keep a thousand complex, interdependent, interacting, sometimes conflicting things sane and working. >Point : The fact that SCMs are two way is great in > a production environment. No packaging solution > can come close. Try talking with some hard-core operations guys, the folks with the keys to the data centers, who work tireless, insanely hours keeping incredibly complex systems running with very little downtime. I think you'd get a different perspective to put it mildly. :) to-sleep-perchance-to-dream-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 06:10:15 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 16:10:15 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <87wrzdfm1y.fsf@benfinney.id.au> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> Message-ID: <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> > "David Lyon" writes: > >> Being honest, if wonderful libraries like Sphinx and Mercurial and Git >> and BZR can't make it into the stdlib, then there is no hope for even >> newer code to get in there. > > Those are applications, not libraries. Applications don't belong in the > standard library. Haha funny.. Well using that logic, distutils is an application.. Are you saying that distutils should be removed? That is most certainly an application. Lets not get too pedantic here. Mercurial and bzr have a built in API that can be called in a library like way. It's true they also have a command line interface in the same way that distutils does. I'm not saying anything negative about distutils. Given that Tarek has an upcoming Pycon presentation where the program talks about a distutils revamp. I'm hoping that he can find some young 20 yr olds and put a cool web interface on that thing. Given that there are empty sprints at pycon. It couldn't hurt to throw that challenge out. Anyway, we'll see.. David From ben+python at benfinney.id.au Wed Jan 20 06:32:10 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:32:10 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> <20100119235036.57f6e867@freewill> Message-ID: <87iqaxfijp.fsf@benfinney.id.au> Barry Warsaw writes: > On Jan 20, 2010, at 03:29 PM, David Lyon wrote: > >So here I am as a user doing things the 'right' way. > > Actually, you're not. It's not up to the Python community to initiate > this. If you really want this, you should engage with the relevant > DVCS communities and push them to request it. Where ?push? must be strictly limited by a continual awareness that the whole idea could just be bad. If you find yourself in a tiny minority pushing for a change, it *could* be that you have a great idea and the vast majority don't realise it yet. But you must be realistic about the likelihood that the change is a very *bad* idea, and frequently evaluate it for signs of that. -- \ ?I used to think that the brain was the most wonderful organ in | `\ my body. Then I realized who was telling me this.? ?Emo Philips | _o__) | Ben Finney From ben+python at benfinney.id.au Wed Jan 20 06:34:07 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:34:07 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> Message-ID: <87eillfigg.fsf@benfinney.id.au> "David Lyon" writes: > Well using that logic, distutils is an application.. Distutils is an application, the function of which is essential to allowing sane development of Python packages. It's a special case. We need to strictly limit the number of special cases, not gleefully add to them. -- \ ?I'd take the awe of understanding over the awe of ignorance | `\ any day.? ?Douglas Adams | _o__) | Ben Finney From matthieu.brucher at gmail.com Wed Jan 20 07:49:36 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 07:49:36 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. I think there are several points that make them not includable in Python: - git is not written in Python - bzr and mercurial have a life cycle much shorter than Python's, it's the same issue than with other libraries where another community develops them. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From david.lyon at pythontest.org Wed Jan 20 08:20:02 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 18:20:02 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Matthieu, >> I'd be happy to see bzr and mercurial and git all made it together >> into the stdlib for python 3. That would give a superb updating >> mechanism for python that would propel python well beyond >> the dinosaur badlands of CPAN and other languages. > > I think there are several points that make them not includable in Python: > - git is not written in Python > - bzr and mercurial have a life cycle much shorter than Python's, it's > the same issue than with other libraries where another community > develops them. That's only two points. :-) On 1; If that's true, I won't mention git again. On 2; Who knows what their life cycle is. CVS is pretty much dead, and svn looks like it is on the way out. I can't think of how anything could be better than mercurial or bzr but I know I will be proved wrong. At the end of the day, we are making a decision about whether the language is 'set-in-stone' or whether it is still evolving. To me, Python 1.x had it's own distinct "era", as has Python 2.x Hoping that the Python 3 era can be a little more flexible and perphaps "cleaner" than the 2.x era is all that I am thinking here. Have a nice day David From matthieu.brucher at gmail.com Wed Jan 20 09:05:19 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 09:05:19 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: > That's only two points. :-) In French, we say that several starts with 2 ;) > On 1; If that's true, I won't mention git again. I tis, you can check on the git repository (it's a mix of C, perl, shell scripts, Python, ...) > On 2; Who knows what their life cycle is. You can check on their websites, their cycles are far shorter than Python minor releases (several months vs several years). Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From abpillai at gmail.com Wed Jan 20 10:36:53 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Wed, 20 Jan 2010 15:06:53 +0530 Subject: [Python-Dev] E3 BEFEHLE In-Reply-To: References: Message-ID: <8548c5f31001200136r54bc9e2aw49485280864ecb2d@mail.gmail.com> On Sun, Jan 17, 2010 at 9:42 AM, Dj Gilcrease wrote: > 2010/1/16 Jack Diederich : > > Good lord, did this make it past other people's spam filters too? I > > especially liked the reference to "REGION -2,0 ; Rlyeh". Ph'nglui > > mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn to you too sir. > > Ya made it past mine too, it looks like a debug dump of a macro for a > some German game based either LOTR or Cthulhu > I initially thought it was a Python disassembler trace of some step of operations which failed, converted to German. In fact, I was looking for a question at the end regarding REPL. How very optimistic... > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Jan 20 11:22:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 20 Jan 2010 19:22:55 +0900 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <87aaw9gjnk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > (Besides, git in the stdlib doesn't make much sense :). "Dulwich." From solipsis at pitrou.net Wed Jan 20 12:28:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Jan 2010 11:28:16 +0000 (UTC) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon pythontest.org> writes: > > I think I'll have better chances with PEPs. > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > [snip] This is python-ideas material, can you take it there? Thank you. Antoine. From ncoghlan at gmail.com Wed Jan 20 13:16:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 22:16:34 +1000 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B56F422.5010607@gmail.com> David Lyon wrote: > On 2; Who knows what their life cycle is. CVS is pretty much > dead, and svn looks like it is on the way out. > I can't think of how anything could be better than > mercurial or bzr but I know I will be proved wrong. I believe you misunderstood what Matthieu meant by life cycle there: think "release cycle". If a project pushes out new releases significantly more often than every 18-24 months (as is currently true for all of the major SCM tools), then that fact alone makes it a very bad fit for the Python standard library. And centralised source control will be going strong for years. The DVCS approach may be great for the open source world, but the gains are far more limited in a closed source shop (especially a group writing internal corporate applications which doesn't need to keep many, if any, maintenance branches going). If we weren't dealing with 4 active branches, the DVCS discussion would have got a lot less traction with the core developers - aside from better handling of multiple lines of development, most of the benefits of the switch to a DVCS accrue to people without commit access to the SVN repository. Anyway, we've wandered far afield from legit python-dev topics now. Any further ideas about super_mega_easy_install functionality that can pull code from source control systems and build it rather than requiring prebuild source tarballs should be directed to python-ideas (they probably need to bake more even before they make an appearance on distutils-sig). Cheers, Nick. P.S. As Jesse said... your enthusiasm is great, but please don't assume that some inherent conservatism on the part of other developers is automatically evil or the result of a failure to see your point. A lot of people around the world rely on our stuff every day. We owe it to them to be measured in our actions and to put serious thought into any major changes or additions we make to the language and the standard library. For the current stage of its development, Python 3 is in a good place from our point of view - its major carrot has really always been the better Unicode support it offers, and the ever-increasing globalisation of the web will create more and more pressure pushing developers in that direction as the years go by. Sure, Python 3 cleans up assorted other things as well, but the change to the text processing model is the big one that is fundamentally incompatible with the architecture of the 2.x series. Compared to that change, everything else is just tinkering. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Fri Jan 1 16:07:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:07:20 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04D81D51@TK5EX14MBXC118.redmond.corp.microsoft.com> <94bdd2610912141458m52da21ear4970506a37214e6@mail.gmail.com> <4dab5f760912230700l38a597f7l855bb2304025d6d5@mail.gmail.com> Message-ID: <94bdd2611001010707h4043ff64x7476049e435852b@mail.gmail.com> 2009/12/23 Glyph Lefkowitz : [..] > 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". > > The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. ?In other words it's got "bin", "lib", "share", "etc", etc.. ?~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation. Right, good idea Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Fri Jan 1 16:32:27 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:32:27 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> Message-ID: <94bdd2611001010732o38a563bcu6d0cc9122ed5c88f@mail.gmail.com> On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziad? wrote: > Hello, > > A while ago I've proposed to refactor the APIs that provides access to > the installation paths and configuration variables at runtime into a > single module called "sysconfig", and make it easier for all > implementations to work with them. I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek From status at bugs.python.org Fri Jan 1 18:07:11 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Jan 2010 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100101170711.A5AAF78654@psf.upfronthosting.co.za> ACTIVITY SUMMARY (12/25/09 - 01/01/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2537 open (+22) / 16902 closed (+19) / 19439 total (+41) Open issues with patches: 1016 Average duration of open issues: 705 days. Median duration of open issues: 462 days. Open Issues Breakdown open 2504 (+22) pending 32 ( +0) Issues Created Or Reopened (42) _______________________________ doc: Code is not always colored 12/30/09 CLOSED http://bugs.python.org/issue7487 reopened flox documention buglet for PyBuffer 12/26/09 CLOSED http://bugs.python.org/issue7577 created ronaldoussoren easy Behavior of operations on a closed file object is not documented 12/26/09 http://bugs.python.org/issue7578 created blep Patch to add docstrings to msvcrt 12/26/09 http://bugs.python.org/issue7579 created brian.curtin patch distutils makefile from python.org installer doesn't work on OS 12/27/09 http://bugs.python.org/issue7580 created bskaplan incomplete doc of zlib 12/27/09 CLOSED http://bugs.python.org/issue7581 created coolwanglu [patch] diff.py to use iso timestamp 12/27/09 http://bugs.python.org/issue7582 created techtonik patch doctest should normalize tabs when comparing output 12/27/09 http://bugs.python.org/issue7583 created techtonik datetime.rfcformat() for Date and Time on the Internet 12/27/09 http://bugs.python.org/issue7584 created techtonik [patch] difflib should separate filename from timestamp with tab 12/27/09 http://bugs.python.org/issue7585 created techtonik patch Typo in collections documentation 12/28/09 CLOSED http://bugs.python.org/issue7586 created nneonneo Python 3.1.1 source build issues 12/28/09 CLOSED http://bugs.python.org/issue7587 created sharmaag unittest.TestCase.shortDescription isn't short anymore 12/28/09 http://bugs.python.org/issue7588 created exarkun setup.py shouldn't try to build nis module when nis headers aren 12/28/09 CLOSED http://bugs.python.org/issue7589 created Arfrever patch 'exceptions' module mentioned in documentation 12/28/09 CLOSED http://bugs.python.org/issue7590 created July test_get_platform fails on 3.1 12/28/09 http://bugs.python.org/issue7591 created pitrou buildbot ssl module documentation: SSLSocket.unwrap description shown twi 12/28/09 http://bugs.python.org/issue7592 created mnewman Computed-goto patch for RE engine 12/29/09 http://bugs.python.org/issue7593 created akuchling patch, needs review shlex refactoring 12/29/09 http://bugs.python.org/issue7594 created ferringb patch, needs review Doc typo for select.kevent() 12/29/09 CLOSED http://bugs.python.org/issue7595 created whit537 test_logging sometimes fails 12/29/09 CLOSED http://bugs.python.org/issue7596 created pitrou curses.use_env implementation error 12/30/09 http://bugs.python.org/issue7597 created kanru patch Cannot install, problems with assembly? 12/30/09 CLOSED http://bugs.python.org/issue7598 created sh.00 (c)ElementTree can produce invalid XML 12/30/09 http://bugs.python.org/issue7599 created jwilk interpreter crashes before start 12/30/09 CLOSED http://bugs.python.org/issue7600 created mhh91 Installation - 2 tests failed: test_cmd_line test_xmlrpc 12/30/09 CLOSED http://bugs.python.org/issue7601 created sadhak Doc: make clean and make update do not delete or update Doc/tool 12/30/09 CLOSED http://bugs.python.org/issue7602 created flox There is no 'seq' type 12/30/09 CLOSED http://bugs.python.org/issue7603 created tom66 delattr __slots__ inconsistancy 12/30/09 CLOSED http://bugs.python.org/issue7604 created ferringb test_cmd_line fails with non-ascii path 12/30/09 http://bugs.python.org/issue7605 created pitrou test_xmlrpc fails with non-ascii path 12/30/09 http://bugs.python.org/issue7606 created pitrou stringlib fastsearch could be improved on 64-bit builds 12/30/09 http://bugs.python.org/issue7607 created pitrou PyUnicode_FromFormatV handles %R and %S incorrectly. 12/30/09 CLOSED http://bugs.python.org/issue7608 created alexandre.vassalotti Add --with-system-expat option 12/31/09 CLOSED http://bugs.python.org/issue7609 created Arfrever patch Cannot use both read and readline method in same ZipExtFile obje 12/31/09 http://bugs.python.org/issue7610 created lucifer shlex not posix compliant when parsing "foo#bar" 12/31/09 http://bugs.python.org/issue7611 created jjdmol2 Fix "pass and object" typo in Library Reference / Built-in Types 12/31/09 CLOSED http://bugs.python.org/issue7612 created ygale patch [cppcheck] found a regression : Invalid number of character ((). 12/31/09 CLOSED http://bugs.python.org/issue7613 created ettl.martin patch Python 2.6.4 segfaults 12/31/09 CLOSED http://bugs.python.org/issue7614 created ttsiod unicode_escape codec does not escape quotes 01/01/10 http://bugs.python.org/issue7615 created rhansen patch test_memoryview test_setitem_writable failures with Intel ICC 01/01/10 http://bugs.python.org/issue7616 created ivank distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option 01/01/10 http://bugs.python.org/issue7617 created Arfrever patch Issues Now Closed (46) ______________________ True division of integers could be more accurate 715 days http://bugs.python.org/issue1811 mark.dickinson patch API to clear most free lists 690 days http://bugs.python.org/issue2019 amaury.forgeotdarc patch unit test UnicodeWarning 687 days http://bugs.python.org/issue2100 ezio.melotti test_disutils fails 568 days http://bugs.python.org/issue3054 ronaldoussoren patch test_formatdate_usegmt fails on non-englisch locale 451 days http://bugs.python.org/issue4046 r.david.murray "??" in u"foo" raises a misleading error 410 days http://bugs.python.org/issue4328 r.david.murray zipfile - add __exit__ attribute to make ZipFile object compatib 287 days http://bugs.python.org/issue5511 ezio.melotti patch test_urllib2 fails - urlopen error file not on local host 271 days http://bugs.python.org/issue5625 orsenthil Improve --with-dbmliborder option 170 days http://bugs.python.org/issue6491 benjamin.peterson patch Move the special-case for integer objects out of PyBytes_FromObj 141 days http://bugs.python.org/issue6687 alexandre.vassalotti patch, 26backport setup.py fails to find headers of system libffi 104 days http://bugs.python.org/issue6943 benjamin.peterson patch, easy The replacement suggested for callable(x) in py3k is not equival 92 days http://bugs.python.org/issue7006 benjamin.peterson patch C/API - Document exceptions 89 days http://bugs.python.org/issue7033 lekma patch subprocess.check_output: "docstring has inconsistent leading whi 35 days http://bugs.python.org/issue7381 georg.brandl patch optparse Documentation References Example Files that do not Exis 30 days http://bugs.python.org/issue7404 georg.brandl patch datetime.datetime.isoformat truncation problem 29 days http://bugs.python.org/issue7413 amaury.forgeotdarc patch, needs review doc: Code is not always colored 0 days http://bugs.python.org/issue7487 georg.brandl float('nan')**2 != nan. float('nan')**2 error 33 on windows 13 days http://bugs.python.org/issue7534 mark.dickinson patch If a generator raises TypeError when being unpacked, an unrelate 10 days http://bugs.python.org/issue7548 amaury.forgeotdarc ctypes doc improvement: c_char_p 6 days http://bugs.python.org/issue7569 georg.brandl Strange issue : cursor.commit() with sqlite 5 days http://bugs.python.org/issue7572 ghaering tes_math fails Mac OS X 10.4 due to OverflowError in test_mtestf 5 days http://bugs.python.org/issue7575 mark.dickinson patch documention buglet for PyBuffer 2 days http://bugs.python.org/issue7577 georg.brandl easy incomplete doc of zlib 0 days http://bugs.python.org/issue7581 amaury.forgeotdarc Typo in collections documentation 0 days http://bugs.python.org/issue7586 georg.brandl Python 3.1.1 source build issues 0 days http://bugs.python.org/issue7587 amaury.forgeotdarc setup.py shouldn't try to build nis module when nis headers aren 1 days http://bugs.python.org/issue7589 benjamin.peterson patch 'exceptions' module mentioned in documentation 1 days http://bugs.python.org/issue7590 georg.brandl Doc typo for select.kevent() 0 days http://bugs.python.org/issue7595 georg.brandl test_logging sometimes fails 1 days http://bugs.python.org/issue7596 ezio.melotti Cannot install, problems with assembly? 0 days http://bugs.python.org/issue7598 ezio.melotti interpreter crashes before start 0 days http://bugs.python.org/issue7600 r.david.murray Installation - 2 tests failed: test_cmd_line test_xmlrpc 1 days http://bugs.python.org/issue7601 ezio.melotti Doc: make clean and make update do not delete or update Doc/tool 0 days http://bugs.python.org/issue7602 georg.brandl There is no 'seq' type 0 days http://bugs.python.org/issue7603 benjamin.peterson delattr __slots__ inconsistancy 0 days http://bugs.python.org/issue7604 benjamin.peterson PyUnicode_FromFormatV handles %R and %S incorrectly. 0 days http://bugs.python.org/issue7608 alexandre.vassalotti Add --with-system-expat option 1 days http://bugs.python.org/issue7609 benjamin.peterson patch Fix "pass and object" typo in Library Reference / Built-in Types 0 days http://bugs.python.org/issue7612 ezio.melotti patch [cppcheck] found a regression : Invalid number of character ((). 0 days http://bugs.python.org/issue7613 ezio.melotti patch Python 2.6.4 segfaults 0 days http://bugs.python.org/issue7614 mark.dickinson Fwd: Debian Bug#42318: urllib.py has problems with malformed pro 3436 days http://bugs.python.org/issue210849 shinnosuke urllib / urllib2 should cache 301 redirections 2425 days http://bugs.python.org/issue735515 pitrou fast modular exponentiation 2084 days http://bugs.python.org/issue936813 mark.dickinson patch bdist_deb - Debian packager 316 days http://bugs.python.org/issue1054967 astraw patch Carbon.Scrap.PutScrapFlavor 987 days http://bugs.python.org/issue1700507 ronaldoussoren Top Issues Most Discussed (10) ______________________________ 11 Add Option to Bind to a Local IP Address in httplib.py 462 days open http://bugs.python.org/issue3972 8 fast modular exponentiation 2084 days closed http://bugs.python.org/issue936813 6 [patch] difflib should separate filename from timestamp with ta 5 days open http://bugs.python.org/issue7585 6 [patch] diff.py to use iso timestamp 5 days open http://bugs.python.org/issue7582 6 Implement fastsearch algorithm for rfind/rindex 24 days open http://bugs.python.org/issue7462 5 test_xmlrpc fails with non-ascii path 2 days open http://bugs.python.org/issue7606 5 test_logging sometimes fails 1 days closed http://bugs.python.org/issue7596 5 Patch to add docstrings to msvcrt 6 days open http://bugs.python.org/issue7579 5 Distutils-based installer does not detect 64bit versions of Pyt 126 days open http://bugs.python.org/issue6792 5 _sha256 et al. encode to UTF-8 by default 17 days open http://bugs.python.org/issue3745 From stefan_ml at behnel.de Sun Jan 3 09:11:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jan 2010 09:11:16 +0100 Subject: [Python-Dev] Providing support files to assist 3.x extension authors In-Reply-To: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> References: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> Message-ID: Case Vanhorsen, 20.12.2009 01:38: > When I ported gmpy (Python to GMP multiple precision library) to > Python 3.x, I began to use PyLong_AsLongAndOverflow frequently. I > found the code to slightly faster and cleaner than using PyLong_AsLong > and checking for overflow. You might want to look at the code Cython generates for integer type conversions. We use specialised coercion code that gets generated on-the-fly to convert Python long/int from and to all sorts of C integer types with compile time (portability) and runtime size/value checks. Depending on your needs, this may or may not be faster than the above C-API function. Stefan From david.lyon at preisshare.net Mon Jan 4 00:42:15 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 10:42:15 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> Message-ID: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> > On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: > > This new operator removes the ambiguity the original proposal had, > without making it more > complex for common use cases. So if you dislike it, you will need to > propose something > else that also fixes the ambiguity we had. Ok. > Environment markers >.. >Here are some example of fields using such markers: > >Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: [Windows] pywin32 1.0+ That's simpler, shorter, and less ambiguous. Easier to parse for package managers. David From python at mrabarnett.plus.com Mon Jan 4 01:14:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 04 Jan 2010 00:14:43 +0000 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4132F3.7020905@mrabarnett.plus.com> David Lyon wrote: >> On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: >> >> This new operator removes the ambiguity the original proposal had, >> without making it more >> complex for common use cases. So if you dislike it, you will need to >> propose something >> else that also fixes the ambiguity we had. > > Ok. > >> Environment markers >> .. >> Here are some example of fields using such markers: >> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. > 'win32' is more specific than 'Windows' and, to me, '1.0+' means '>=1.0', not '>1.0'. From martin at v.loewis.de Mon Jan 4 01:21:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jan 2010 01:21:53 +0100 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4134A1.5090203@v.loewis.de> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. Don't you want the PEP to complete? Why this bike-shedding? I can agree it's shorter. I can't agree that it's simpler, or less ambiguous. Regards, Martin From ssteinerx at gmail.com Mon Jan 4 01:29:14 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 3 Jan 2010 19:29:14 -0500 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: On Jan 3, 2010, at 7:21 PM, Martin v. L?wis wrote: >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Really.... Enough, already! S From david.lyon at preisshare.net Mon Jan 4 01:47:56 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 11:47:56 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: <1349.218.214.45.58.1262566076.squirrel@syd-srv02.ezyreg.com> Hi Martin, Happy New Year, >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Well, I'm just helping out by pointing out some simpler ways as Tarek asked me. I was only answering his question. I was out celebrating so it took longer to reply than normal. Bike-Shedding ? Me ? which bikeshed? wanting simple? Anyway, I'm just reading the PEPs and commenting. If there are some alterations that can be done, lets discuss them. > I can agree it's shorter. > .. Cool. What I'd really like is a 'Code-Repository:' keyword so that we can install programs/libraries directly into a system. I feel that this would really simplify the packaging landscape, making it much easier for the scientific community and others to do python software installs. This would allow us to perphaps have something even *more modern* than CPAN. So yes, getting PEP-345 right is important to me. Have a nice day. David From abpillai at gmail.com Mon Jan 4 10:08:05 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 4 Jan 2010 14:38:05 +0530 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: <8548c5f31001040108v635a78ech33521371c6c50e82@mail.gmail.com> On Sun, Dec 27, 2009 at 11:21 PM, anatoly techtonik wrote: > On Tue, Dec 15, 2009 at 12:37 AM, Steven Bethard > wrote: > > > > If you're only concerned about 2.X, then yes, optparse will *never* be > > removed from 2.X. There will be a deprecation note in the 2.X > > documentation but deprecation warnings will only be issued when the -3 > > flag is specified. Please see the "Deprecation of optparse" section of > > the PEP: > > > > http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Ref http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse . Considering that optparse will be deprecated like 5 years from now. I think this point is moot. The deprecation strategy IMHO is perhaps way too conservative. Maybe it should be deprecated faster ;) -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Mon Jan 4 10:32:10 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 4 Jan 2010 10:32:10 +0100 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: >> If you're only concerned about 2.X, then yes, optparse will *never* be >> removed from 2.X. There will be a deprecation note in the 2.X >> documentation but deprecation warnings will only be issued when the -3 >> flag is specified. Please see the "Deprecation of optparse" section of >> the PEP: >> >> http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Compare: > http://docs.python.org/library/optparse.html > http://argparse.googlecode.com/svn/tags/r101/doc/index.html > > argparse should be recommended as advanced and more featured variant > of optparse - that goes without saying, but forcing people to switch > and annoying them with deprecation messages brings only headache. As has been noted already nobody is forcing people to switch. Optparse will be available as a separate package and everybody will be free to install it and will not have any deprecation messages anywhere. > Just > like optparse is better getopt, the latter could also be useful for > people coming from other languages and porting their libraries to > Python. > > I would better concentrate on real code examples how argparse solves > problems and would really want to see > http://www.python.org/dev/peps/pep-0389/#out-of-scope-various-feature-requests > implemented until argparse enters phase where backward incompatible > changes in API are impossible. > > I would prefer to see PEP 389 as a document describing proposed > solutions to argument parsing problems rather than petition to replace > one library with another. So, it should display common argument > parsing scenarios (use cases) with examples that are also useful > recipes for documentation. I guess more than 90% people here doesn't > have time to read argparse methods descriptions to see what they could > be useful for. -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From olemis at gmail.com Mon Jan 4 15:24:05 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 4 Jan 2010 09:24:05 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> Message-ID: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: > Thanks for the quick response. > > On 30/12/2009, Benjamin Peterson wrote: >> >> When I made that change, I didn't know that the __unittest "hack" was >> being used elsewhere outside of unittest, so I felt fine replacing it >> with another. While I still consider it an implementation detail, I >> would be ok with exposing an "official" API for this. Perhaps >> __unittest_ignore_traceback? > > Well, bazaar has had the trick for a couple of years, and googling > around now turns up some other projects using it or thinking about it: > > > > Add `dutest` and probably `nose` [2]_ and ... > Reinstating the old implementation (with the same name) would mean > that existing code would keep working with Python 2.7 IOW ... if it is removed all the libraries will have to change and check for the Py version and ... (probably not a big deal depending on the details of the ?solution?) > but maybe a > discussion could start about a new, less hacky, way of doing the same > I am strongly -1 for modifying the classes in ?traditional? unittest module [2]_ (except that I am strongly +1 for the package structure WITHOUT TOUCHING anything else ...) , and the more I think about it I am more convinced ... but anyway, this not a big deal (and in the end what I think is not that relevant either ... :o). So ... pass > May not be worthwhile making life more complicated though, > there aren't *that* many unittest-extending projects. > But every library extending `unittest` will do it or otherwise not-so-useful error messages will be displayed ;o) PS: Happy New Year ! BTW .. [1] [feature] nosexml should omit trace frames where `__unittest... (http://code.google.com/p/python-nosexml/issues/detail?id=4#c0) .. [2] Assessment of unittest 2.7 API : new features and opinions (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free new ripit 1.3.3 Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/KFwyUTKH0vI/ From juanfhj at gmail.com Tue Jan 5 17:10:16 2010 From: juanfhj at gmail.com (Juan Fernando Herrera J.) Date: Tue, 5 Jan 2010 11:10:16 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility Message-ID: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Jan 5 17:50:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 05 Jan 2010 16:50:15 +0000 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B436DC7.8040605@voidspace.org.uk> On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way > major software hasn't been ported to it. I'm eager to use 3, but > paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me > if this has been suggested in the past. > I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well. I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter). All the best, Michael Foord > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Jan 5 18:21:31 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 5 Jan 2010 11:21:31 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion. I'm sure everyone out there would like to dive into supporting 3.x, but it's tough to prioritize when you are up to your eyeballs with 2.x bugs in your tracker. Look at Twisted ( http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3) -- over 1000 issues, ~5 developers -- 3.x support won't be here tomorrow, but it's on the horizon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 18:23:08 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2010 09:23:08 -0800 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B436DC7.8040605@voidspace.org.uk> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B436DC7.8040605@voidspace.org.uk> Message-ID: On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord wrote: > On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > > I don't know about other developers, but I certainly expected Python 3 > adoption to take a few years due to libraries only gradually making the > jump. I also expected there to be nervousness and pain around the migration > as well. > > I think in fact that libraries *are* migrating and there are lots of > encouraging signs. As a community we need to do all we can to promote Python > 3 adoption, but I don't think there is much reason to be worried (or > complacent for that matter). > Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Jan 5 18:52:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 18:52:20 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <319e029f1001050952o71cb29afh66445550beeb99c2@mail.gmail.com> On Tue, Jan 5, 2010 at 17:10, Juan Fernando Herrera J. wrote: > I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. Yes. Python 3 is not what you want to use today if you write applications. If you write libraries 2010 is the year to port, IMO. With some luck, 2011 will be the year to start porting applications properly. We'll see. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ianb at colorstudy.com Tue Jan 5 20:00:49 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 13:00:49 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin wrote: > On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > >> How about a new python 3 release with (possibly partial) backwards >> compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major >> software hasn't been ported to it. I'm eager to use 3, but paradoxically, >> the 3 release makes me rather stuck with 2.6. Excuse me if this has been >> suggested in the past. >> >> > The proper route to take, in my opinion, is to see what 2.x libraries you > are using that are not 3.x compatible, run 2to3 on them, then run their test > suite, and see where you get. Submit a patch or two to the library and see > what happens -- it at least gets the wheels in motion. > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 5 21:24:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 5 Jan 2010 15:24:45 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> On Jan 5, 2010, at 2:00 PM, Ian Bicking wrote: > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. It seems like this is a problem to be addressed, then. Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. > The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). No-op constructions like 'bytes("")' could help for older versions of Python, though. A very, very small runtime shim could provide support for these, if 2to3 could be told about it somehow. > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? Sort of like our own version of the #line directive? :) Seriously though, I find it hard to believe that this is a big problem. The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. > I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. This is almost certainly more of a pain than trying to trick 2to3 into doing the right thing. From regebro at gmail.com Tue Jan 5 21:57:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 21:57:35 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <319e029f1001051257k3827c52ahcd115b15d9a8ece7@mail.gmail.com> On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz wrote: > It seems like this is a problem to be addressed, then. ?Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x. >> The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated. > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 5 22:07:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 22:07:23 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B43AA0B.5030308@v.loewis.de> > It's not even that easy -- libraries can't apply patches for Python 3 > compatibility as they usually break Python 2 compatibility. Potentially > libraries could apply patches that make a codebase 2to3 ready, but from > what I've seen that's more black magic than straight forward updating, > as such patches have to trick 2to3 producing the output that is desired. I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x. > The only workable workflow I've seen people propose for maintaining a > single codebase with compatibility across both 2 and 3 is to use such > tricks, with aliases to suppress some 2to3 updates when they are > inappropriate I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way. > Also, running 2to3 on installation is kind of annoying, as you get > source that isn't itself the canonical source, so to fix bugs you have > to look at the installed source and trace it back to the bug in the > original source. That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin From martin at v.loewis.de Tue Jan 5 22:13:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jan 2010 22:13:07 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <4B43AB63.3000607@v.loewis.de> > No-op constructions like 'bytes("")' could help for older versions of > Python, though. A very, very small runtime shim could provide > support for these, if 2to3 could be told about it somehow. You actually don't *need* 2to3 support for that - bytes("") can work in either version: 2.x: def bytes(s): return s 3.x: def bytes(s) return s.encode("ascii") On top of that, you *could* as for bytes("") to be converted to b"" in 3.x - and it is indeed possible to tell 2to3 about that, and you don't even need to modify 2to3's source to do so: it can be part of your package. >> Also, running 2to3 on installation is kind of annoying, as you get >> source that isn't itself the canonical source, so to fix bugs you >> have to look at the installed source and trace it back to the bug >> in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in > .pycs rather than based on where the code was actually loaded in the > filesystem, couldn't 2to3 could do .pyc rewriting to point at the > original source? Sort of like our own version of the #line > directive? :) I think it could, but it would be fairly flaky as the pycs can get lost, and lose that information after regeneration. Also, it may be confusing in other scenarios. I'd rather have it generate separate mapper files, and change the traceback printing to consider these (as an option). > Seriously though, I find it hard to believe that this is a big > problem. The 3.x source looks pretty similar to the 2.x source, and > it's good to look at both if you're dealing with a 3.x issue. That's my experience as well - the only challenge is that you can't cut-n-paste the source URL into an editor. Regards, Martin From ianb at colorstudy.com Tue Jan 5 22:39:21 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 15:39:21 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43AA0B.5030308@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: On Tue, Jan 5, 2010 at 3:07 PM, "Martin v. L?wis" wrote: > > > It's not even that easy -- libraries can't apply patches for Python 3 > > compatibility as they usually break Python 2 compatibility. ?Potentially > > libraries could apply patches that make a codebase 2to3 ready, but from > > what I've seen that's more black magic than straight forward updating, > > as such patches have to trick 2to3 producing the output that is desired. > > I wouldn't qualify it in that way. It may be necessary occasionally to > trick 2to3, but that's really a bug in 2to3 which you should report, so > that trickery is then a work-around for a bug - something that you may > have to do with other API, as well. > > The "black magic" is really more in the parts that 2to3 doesn't touch > at all (because they are inherently not syntactic); these are the > problem areas Guido refers to. The "black magic" then is to make the > same code work unmodified for both 2.x and 3.x. Just to clarify, the black magic I'm referring to is things like: try: ?? ?unicode_ = unicode except NameError: ?? ?unicode_ = str and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). ?If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. ?It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. ?For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str. ?That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself. Also replying to Glyph: > > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? ?Sort of like our own version of the #line directive? :) > > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun. So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits. Which is a terribly crotchety of me. Sorry. -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From martin at v.loewis.de Tue Jan 5 23:23:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 23:23:53 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: <4B43BBF9.2000302@v.loewis.de> > Just to clarify, the black magic I'm referring to is things like: > > try: > unicode_ = unicode > except NameError: > unicode_ = str > > and some other aliases like this that are unambiguous and which 2to3 > won't touch (if you write them correctly). If the porting guide noted > all these tricks (of which several have been developed, and I'm only > vaguely aware of a few) that would be helpful. It's not a lot of > tricks, but the tricks are not obvious and 2to3 gets the translation > wrong pretty often without them. For instance, when I say str in > Python 2 I often means bytes, unsurprisingly, but 2to3 translates both > str and unicode to str. No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3? > That *nothing* translates to bytes by default > (AFAIK) means that people must either be living in a bytes-free world > (which sure, lots of code does) or they are using tricks not included > in 2to3 itself. By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3). > Since 2to3 maintains line numbers yes, it wouldn't be that bad. But > then I don't currently develop any code that is "installed", I only > develop code that is directly from a source code checkout, and where > the checkout is put on the path. I guess I could have something that > automatically builds the code on every edit, and that's not > infeasible. It's just not fun. Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it. > So long as I have to support Python 2 > (which is like forever) then adding Python 3 only makes development > that much more complicated and much less fun, with no concrete > benefits. I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin From ziade.tarek at gmail.com Wed Jan 6 01:08:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 01:08:34 +0100 Subject: [Python-Dev] PEP 386 and PEP 345 Message-ID: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Hi, I think we've reached a consensus on those two PEPs. Although, there's one last point that was forgotten in the discussions : I've introduced "rc" in the pre-releases markers, so PEP 386 is compatible with Python's own version scheme. "rc" comes right after "c" in the sorting. It's slightly redundant with the "c" marker but I don't think this really matters as long as consumers know how to order them (a < b < c < rc). I have also stated that "c" is the preferred marker for third party projects, from PEP 386 point of view. Is there anything else I can do to make those two PEPs accepted ? Regards Tarek -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Wed Jan 6 01:26:34 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:26:34 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43BBF9.2000302@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> <4B43BBF9.2000302@v.loewis.de> Message-ID: <1322.218.214.45.58.1262737594.squirrel@syd-srv02.ezyreg.com> Hi Martin, > ... but in the long run, starting now will have saved you a lot of > pressure in the long run, and stop users from switching away from > your packages because of lack of Python 3 support. In a production situation it works the other way around. If there's an application that requires twisted (or whatever package) then most people would use the appropriate interpreter to match the library. Since you guys all did your jobs so well :-) doing so is painless. Because there is so much "comfort" with the existing situation it makes it an effort for people to move to a different lounge chair. Namely python 3. I'd suggest that moving the package set (pypi) to python 3 could be kicked along with the help of some automated tools. I don't know what tools you guys have got. But I am very sure that if code analysis was provided to package developers on python 3 (so they don't have to run it themselves), then it would be like an even bigger tv screen in a bigger lounge room and it would assist in drawing them over. David From david.lyon at preisshare.net Wed Jan 6 01:36:24 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:36:24 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <1337.218.214.45.58.1262738184.squirrel@syd-srv02.ezyreg.com> Hi Ian, > The only workable workflow I've seen people propose for maintaining a > single > codebase with compatibility across both 2 and 3 is to use such tricks, > with > aliases to suppress some 2to3 updates when they are inappropriate, so that > you can run 2to3 on install and have a single canonical Python 2 source. > Python 2.7 won't help much (even though it is trying) as the introduction > of non-ambiguous constructions like b"" aren't compatible with previous > versions of Python and so can't be used in many libraries (support at > least > back to Python 2.5 is the norm for most libraries, I think). > > Also, running 2to3 on installation is kind of annoying, as you get source > that isn't itself the canonical source, so to fix bugs you have to look at > the installed source and trace it back to the bug in the original source. > > I suspect a reasonable workflow might be possible with hg and maybe patch > queues, but I don't feel familiar enough with those tools to map that out. That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David From david.lyon at preisshare.net Wed Jan 6 03:01:22 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 13:01:22 +1100 (EST) Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: <1617.218.214.45.58.1262743282.squirrel@syd-srv02.ezyreg.com> > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? Tarek, Given that I helped out so much last year with the PEP in discussing different options, even if they weren't accepted, I really feel that it is unfair if my name isn't mentioned. It was a huge time sacrifice on my part. For example, even if I only managed to explain the version numbering and clarify how that worked. It did take me some time to do that. What I did do however, was spend a lot of time with the multiplatform "Markers". I still think that two short weeks more of "discussion" could resolve some issues. That discussion went for 4 months on distutils-ml. Look, major issues aside, can you make the following concessions on PEP-345 which I only feel will help it, namely: 1) Source-Repository: specify a code repository to install from 2) Streamline Requires-Python: by implementing "Markers" as noted by the PEP. A marker being something like "Requires-Python(windows): lxml". Otherwise remove the word marker from the PEP and just replace with "metacode". Markers are what were discussed on distutils-ml. Metacode is what is described in the PEP. 3) Remove the inconsistency and platform ambiguities. Especially for windows users. For example, "win32" is extremely confusing for windows users right now. As more and more systems now are 64 bit. Use the platform module, instead of the sys module for constants. I'll post to distutils-ml on this. I am certainly not trying to hold this PEP up, and I apologise on my part for my late attention. I will post to distutils-ml on these and i promise to keep my comments unheated and unwitty. Having said that, PEP-345 is *super-important*. A week or two or three more discussion and the issues can be resolved. We all just want to focus on being productive. It would be a great accomplishment for you to get PEP-345 approved and likewise for me getting mentioned even in a minor role as helping out on a PEP. So I'm hoping that you can make a few last minute concessions meaning that we can all happily go on our way in 2010. Regards David From brett at python.org Wed Jan 6 06:20:30 2010 From: brett at python.org (Brett Cannon) Date: Tue, 5 Jan 2010 21:20:30 -0800 Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 16:08, Tarek Ziad? wrote: > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? > As you said, consensus has been reached, so just Guido's BDFL stamp of approval is all I can think of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Jan 6 12:19:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:19:45 +0000 Subject: [Python-Dev] bug triage Message-ID: <4B4471D1.9020707@simplistix.co.uk> Hi All, Is there a high volume of incoming bugs to the Python tracker? If so, I'd like to help with triaging. I think I have all the necessary access, what I'm missing is the knowledge of how to set myself up to get notifications of new bugs... How do I do that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at voidspace.org.uk Wed Jan 6 12:24:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 06 Jan 2010 11:24:37 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: <4B4472F5.8000709@voidspace.org.uk> On 06/01/2010 11:19, Chris Withers wrote: > Hi All, > > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the > necessary access, what I'm missing is the knowledge of how to set > myself up to get notifications of new bugs... > > How do I do that? > Bug triaging is one of Python's "big needs" and anything you do to help on this score would be much appreciated. Particularly reviewing new and outstanding issues. I assumed there would be RSS feeds for bug tracker activity but can't easily find these on the tracker. There is a bot that posts activity to #python-dev, so there must be some way of getting this information. All the best, Michael > cheers, > > Chris > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Jan 6 12:25:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 11:25:16 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: Hi Chris, > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Do you really want to get such notifications? There may be a lot of them. If you want however, you can join #python-dev on IRC (irc.freenode.net) where there's a bot which posts updates of all bugs on the tracker. There's usually not a lot of discussion going on so you probably won't feel flooded. In addition to bug triage, what is needed is reviewing of existing patches, as well as writing patches for issues which haven't been addressed yet :-) Regards Antoine. From chris at simplistix.co.uk Wed Jan 6 12:30:40 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:30:40 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B447460.7080100@simplistix.co.uk> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. Yeah, email-out is what I'm really after... I have it for my own Roundup instance so it can't be that hard to do ;-) Roch?, you guys host the bug tracker, right? Is there email-out set up for it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Jan 6 12:37:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 21:37:23 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B4475F3.5040406@gmail.com> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. I'm pretty sure the bugs list is still the primary spooled notification mechanism: http://mail.python.org/mailman/listinfo/python-bugs-list There are also the weekly tracker activity summaries that are posted here to python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From chris at simplistix.co.uk Wed Jan 6 12:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:41:28 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <4B4476E8.8050709@simplistix.co.uk> Nick Coghlan wrote: > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list That's what I was after, thanks! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From facundobatista at gmail.com Wed Jan 6 13:03:08 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 6 Jan 2010 09:03:08 -0300 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: On Wed, Jan 6, 2010 at 8:19 AM, Chris Withers wrote: > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Not notifications, but maybe a way to have a higher look of them for easy selection: http://www.taniquetil.com.ar/cgi-bin/pytickets.py Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From ziade.tarek at gmail.com Wed Jan 6 13:29:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:29:58 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord wrote: > On 06/01/2010 11:19, Chris Withers wrote: >> >> Hi All, >> >> Is there a high volume of incoming bugs to the Python tracker? >> If so, I'd like to help with triaging. I think I have all the necessary >> access, what I'm missing is the knowledge of how to set myself up to get >> notifications of new bugs... >> >> How do I do that? >> > > Bug triaging is one of Python's "big needs" and anything you do to help on > this score would be much appreciated. Particularly reviewing new and > outstanding issues. Another useful triage I think, is to review the oldest bugs (some of them are > 5 years) and remove the ones that are not relevant anymore, or duplicate with newer entries. Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Wed Jan 6 13:31:08 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 12:31:08 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B44828C.4070201@simplistix.co.uk> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with someone as a paired task for those 2 days... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Jan 6 13:37:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:37:55 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B44828C.4070201@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B44828C.4070201@simplistix.co.uk> Message-ID: <94bdd2611001060437s2d0242aembc669c3263773fea@mail.gmail.com> On Wed, Jan 6, 2010 at 1:31 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Another useful triage I think, is to review the oldest bugs (some of >> them are > 5 years) >> and remove the ones that are not relevant anymore, or duplicate with >> newer entries. > > I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with > someone as a paired task for those 2 days... I'll be doing Distutils stuff but I can probably help around a bit in that task: Distutils have quite a few old issues so I can tackle those From roche at upfrontsystems.co.za Wed Jan 6 13:32:39 2010 From: roche at upfrontsystems.co.za (=?ISO-8859-1?Q?Roch=E9?= Compaan) Date: Wed, 06 Jan 2010 14:32:39 +0200 Subject: [Python-Dev] bug triage In-Reply-To: <4B447460.7080100@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B447460.7080100@simplistix.co.uk> Message-ID: <1262781159.2836.218.camel@didi> On Wed, 2010-01-06 at 11:30 +0000, Chris Withers wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > Yeah, email-out is what I'm really after... I have it for my own Roundup > instance so it can't be that hard to do ;-) > > Roch?, you guys host the bug tracker, right? Is there email-out set up > for it? We do, but we don't administer it. There are a few administrators taking care of it and you should be able to reach them by logging your request here: http://psf.upfronthosting.co.za/roundup/meta/ or post it to the Infrastructure mailing list: infrastructure at python.org -- Roch? Compaan Upfront Systems http://www.upfrontsystems.co.za From ncoghlan at gmail.com Wed Jan 6 13:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 22:57:24 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B4488B4.2070208@gmail.com> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I believe someone (Daniel Diniz, maybe?) did do a pass over those some time in the last 12 months, so most of the obviously irrelevant ones that are that old should already be gone. Not to say it isn't worth doing another pass, just saying not to get disheartened if there aren't many that can be readily closed. There are at least a few still kicking around just because they're difficult to deal with (there's an ancient one to do with one of the ways circular imports can fail that I occasionally go back and reread before moving on to something more tractable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Wed Jan 6 14:43:17 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 08:43:17 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <4B4476E8.8050709@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> <4B4476E8.8050709@simplistix.co.uk> Message-ID: <20100106134317.3EF141FB5A6@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:41:28 +0000, Chris Withers wrote: > Nick Coghlan wrote: > > I'm pretty sure the bugs list is still the primary spooled notification > > mechanism: > > http://mail.python.org/mailman/listinfo/python-bugs-list > > That's what I was after, thanks! Just for completeness, there's also new-bugs-announce if you want just *new* bug notification. That's more for people who want to watch for bugs they want to become nosy on, though; if you are doing triage python-bugs-list is what you want. Please also read http://www.python.org/dev/workflow/ if you haven't already. Thanks for being willing to chip in! --David From brian.curtin at gmail.com Wed Jan 6 15:57:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 08:57:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4488B4.2070208@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > I believe someone (Daniel Diniz, maybe?) did do a pass over those some > time in the last 12 months, so most of the obviously irrelevant ones > that are that old should already be gone. Not to say it isn't worth > doing another pass, just saying not to get disheartened if there aren't > many that can be readily closed. > > There are at least a few still kicking around just because they're > difficult to deal with (there's an ancient one to do with one of the > ways circular imports can fail that I occasionally go back and reread > before moving on to something more tractable). > > Cheers, > Nick. > On the topic of bugs that can be readily closed (literally), I've recently come across a number of issues which appear to be sitting in a patch or review stage, but their patches have been committed and the issue remains open. What is the best course of action there? I'd just go ahead and close the issue myself but I don't have tracker privileges. I'm willing to help out with another Daniel Diniz-esque triage sweep if that would help. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Wed Jan 6 16:14:20 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 6 Jan 2010 10:14:20 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <7FCF8B19-3465-4392-80CC-BEAEBD926ECA@gmail.com> On Jan 6, 2010, at 7:29 AM, Tarek Ziad? wrote: > On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord > wrote: >> On 06/01/2010 11:19, Chris Withers wrote: >>> >>> Hi All, >>> >>> Is there a high volume of incoming bugs to the Python tracker? >>> If so, I'd like to help with triaging. I think I have all the necessary >>> access, what I'm missing is the knowledge of how to set myself up to get >>> notifications of new bugs... >>> >>> How do I do that? >>> >> >> Bug triaging is one of Python's "big needs" and anything you do to help on >> this score would be much appreciated. Particularly reviewing new and >> outstanding issues. > > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I was actually thinking about that the other day when I saw that the average age of bugs on the Python tracker was at some hideously large 3 digit number. The 'success' statistic would be to bring that down below, say, 100. S From skip at pobox.com Wed Jan 6 17:38:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 6 Jan 2010 10:38:13 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <19268.48245.616046.874126@montanaro.dyndns.org> >>>>> "Nick" == Nick Coghlan writes: Nick> I'm pretty sure the bugs list is still the primary spooled Nick> notification mechanism: Nick> http://mail.python.org/mailman/listinfo/python-bugs-list Actually, there is a new-bugs-announce list: http://mail.python.org/mailman/listinfo/new-bugs-announce Skip From solipsis at pitrou.net Wed Jan 6 18:56:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 17:56:49 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > On the topic of bugs that can be readily closed (literally), I've > recently come across a number of issues which appear to be sitting in a > patch or review stage, but their patches have been committed and the > issue remains open. What is the best course of action there? Post a message on the issue asking for info. From solipsis at pitrou.net Wed Jan 6 19:09:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 18:09:39 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > > On the topic of bugs that can be readily closed (literally), I've > > recently come across a number of issues which appear to be sitting in a > > patch or review stage, but their patches have been committed and the > > issue remains open. What is the best course of action there? > > Post a message on the issue asking for info. Ok, I realize my answer might have been a bit terse :-) The patch might be waiting to be merged in all development branches, or it may not totally resolve the issue, or perhaps documentation needs to be updated, or perhaps it is pending a verdict from the buildbots, etc. You can't deduce that the issue is completely fixed from the simple fact that something has been committed. Regards Antoine Pitrou. From brett at python.org Wed Jan 6 20:03:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 11:03:32 -0800 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > >> I believe someone (Daniel Diniz, maybe?) did do a pass over those some >> time in the last 12 months, so most of the obviously irrelevant ones >> that are that old should already be gone. Not to say it isn't worth >> doing another pass, just saying not to get disheartened if there aren't >> many that can be readily closed. >> >> There are at least a few still kicking around just because they're >> difficult to deal with (there's an ancient one to do with one of the >> ways circular imports can fail that I occasionally go back and reread >> before moving on to something more tractable). >> >> Cheers, >> Nick. >> > > On the topic of bugs that can be readily closed (literally), I've recently > come across a number of issues which appear to be sitting in a patch or > review stage, but their patches have been committed and the issue remains > open. What is the best course of action there? I'd just go ahead and close > the issue myself but I don't have tracker privileges. > > If a core developer is willing to step forward and vouch for you to get tracker privileges then I will give them to you. We are trying to give out tracker privs w/ less time than required to get commit privileges. So as long as you have helped out on a few issues in a positive and correct way that should be enough to get one of the regulars who perform triage to notice. -Brett I'm willing to help out with another Daniel Diniz-esque triage sweep if that > would help. > > Brian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Wed Jan 6 21:41:05 2010 From: nad at acm.org (Ned Deily) Date: Wed, 06 Jan 2010 12:41:05 -0800 Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: In article <4B4475F3.5040406 at gmail.com>, Nick Coghlan wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list Also, that mailing list (along with most python development related mailing lists) is mirrored at gmane.org which means it can also be obtained via a newsreader (NNTP) or various RSS feeds. http://dir.gmane.org/gmane.comp.python.bugs -- Ned Deily, nad at acm.org From nick.bastin at gmail.com Wed Jan 6 22:14:54 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:14:54 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? Message-ID: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> (This may occur on more platforms - I can test on more unix platforms if the consensus is this is an actual problem and I'm not just a nut) On freebsd5, if you do a simple ./configure --enable-shared in current (2.7) trunk, your python shared library will build properly, but all modules will fail to find the shared library and thus fail to build: gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.7/u1/Python/Python-2.7a1/Modules/_struct.o -L/u1/tmp/python2.7a1/lib -L/usr/local/lib -lpython2.7 -o build/lib.freebsd-5.3-RELEASE-i386-2.7/_struct.so /usr/bin/ld: cannot find -lpython2.7 building '_ctypes_test' extension ... This of course is because libpython2.7.so is in the current directory and not (yet) installed in /usr/local/lib. I've made a very simple fix for this problem that works, but at least to me smells a bit funny, which is to modify setup.py to add the following to detect_modules(): # If we did --enable-shared, we need to be able to find the library # we just built in order to build the modules. if platform == 'freebsd5': add_dir_to_list(self.compiler_obj.library_dirs, '.') Which brings me to a few questions: a) Does this seem like a real problem, or am I missing something obvious? b) Does this fix seem like the sensible thing to do? (it seems at least that we ought to check that the user configured --enable-shared and only set -L. in that case, if that's possible) Setting --enable-shared when you actually have a libpython2.7.so in /usr/local/lib (or whatever --prefix you've selected) is possibly even more dangerous, because it may succeed in linking against a differently-built library than what you intended. -- Nick From nick.bastin at gmail.com Wed Jan 6 22:39:17 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:39:17 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <66d0a6e11001061339t38202b70mca1091e1926ecf4b@mail.gmail.com> On Wed, Jan 6, 2010 at 16:14, Nicholas Bastin wrote: > This of course is because libpython2.7.so is in the current directory > and not (yet) installed in /usr/local/lib. One minor correction - as you could see from the compile line, the actual --prefix in this case is /u1/tmp/python2.7a1, but the libraries obviously aren't installed there yet either. Perhaps a better fix than setting -L. would be to put the shared library in build/lib.freebsd-5.3-RELEASE-i386-2.7 and add that to the library path for the linker (the build creates this directory, but installs nothing in it). -- Nick From martin at v.loewis.de Wed Jan 6 23:21:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jan 2010 23:21:44 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <4B450CF8.8090009@v.loewis.de> > b) Does this fix seem like the sensible thing to do? No. Linking in setup.py should use the same options as if the module was built as *shared* through Modules/Setup, which, IIUC, should use BLDLIBRARY. Regards, Martin From nick.bastin at gmail.com Thu Jan 7 01:08:13 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 19:08:13 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B450CF8.8090009@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> Message-ID: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> On Wed, Jan 6, 2010 at 17:21, "Martin v. L?wis" wrote: >> b) Does this fix seem like the sensible thing to do? > > No. Linking in setup.py should use the same options as if the module > was built as *shared* through Modules/Setup, which, IIUC, should use > BLDLIBRARY. Thanks for that pointer, that makes much more sense. Indeed, BLDLIBRARY on FreeBSD* is set to '-L. -lpython$(VERSION)' if you set --enable-shared, but somehow that piece of information doesn't propagate into the module build. More investigation to be done... -- Nick From rdmurray at bitdance.com Thu Jan 7 02:22:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 20:22:42 -0500 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > > On the topic of bugs that can be readily closed (literally), I've recently > > come across a number of issues which appear to be sitting in a patch or > > review stage, but their patches have been committed and the issue remains > > open. What is the best course of action there? I'd just go ahead and close > > the issue myself but I don't have tracker privileges. > > > > > If a core developer is willing to step forward and vouch for you to get > tracker privileges then I will give them to you. We are trying to give out > tracker privs w/ less time than required to get commit privileges. So as > long as you have helped out on a few issues in a positive and correct way > that should be enough to get one of the regulars who perform triage to > notice. > > -Brett I've done a quick scan of issues Brian is nosy on to refresh my memory, and I'd say he's definitely been making positive contributions. I'm willing to volunteer to keep an eye on his triage work for a while if you grant him tracker privs. Brian, I assume you'll be cognizant of Antoine's advice about making sure a bug really should be closed before closing it :) Hanging out in #python-dev on freenode while working on issues can be helpful, as well, since you can quickly ask whoever is there for second opinions on particular bugs. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Thu Jan 7 02:28:26 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 17:28:26 -0800 Subject: [Python-Dev] bug triage In-Reply-To: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > > On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > > On Wed, Jan 6, 2010 at 06:57, Brian Curtin > wrote: > > > On the topic of bugs that can be readily closed (literally), I've > recently > > > come across a number of issues which appear to be sitting in a patch or > > > review stage, but their patches have been committed and the issue > remains > > > open. What is the best course of action there? I'd just go ahead and > close > > > the issue myself but I don't have tracker privileges. > > > > > > > > If a core developer is willing to step forward and vouch for you to get > > tracker privileges then I will give them to you. We are trying to give > out > > tracker privs w/ less time than required to get commit privileges. So as > > long as you have helped out on a few issues in a positive and correct way > > that should be enough to get one of the regulars who perform triage to > > notice. > > > > -Brett > > I've done a quick scan of issues Brian is nosy on to refresh my > memory, and I'd say he's definitely been making positive contributions. > I'm willing to volunteer to keep an eye on his triage work for a while > if you grant him tracker privs. > > Done for the username brian.curtin (email doesn't match the one Brian emailed from so do let me know, Brian if this is the right username). Welcome aboard! -Brett > Brian, I assume you'll be cognizant of Antoine's advice about making > sure a bug really should be closed before closing it :) Hanging out in > #python-dev on freenode while working on issues can be helpful, as well, > since you can quickly ask whoever is there for second opinions on > particular bugs. > > -- > R. David Murray www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Jan 7 02:59:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 19:59:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 19:28, Brett Cannon wrote: > > > On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > >> >> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: >> > On Wed, Jan 6, 2010 at 06:57, Brian Curtin >> wrote: >> > > On the topic of bugs that can be readily closed (literally), I've >> recently >> > > come across a number of issues which appear to be sitting in a patch >> or >> > > review stage, but their patches have been committed and the issue >> remains >> > > open. What is the best course of action there? I'd just go ahead and >> close >> > > the issue myself but I don't have tracker privileges. >> > > >> > > >> > If a core developer is willing to step forward and vouch for you to get >> > tracker privileges then I will give them to you. We are trying to give >> out >> > tracker privs w/ less time than required to get commit privileges. So as >> > long as you have helped out on a few issues in a positive and correct >> way >> > that should be enough to get one of the regulars who perform triage to >> > notice. >> > >> > -Brett >> >> I've done a quick scan of issues Brian is nosy on to refresh my >> memory, and I'd say he's definitely been making positive contributions. >> I'm willing to volunteer to keep an eye on his triage work for a while >> if you grant him tracker privs. >> >> > Done for the username brian.curtin (email doesn't match the one Brian > emailed from so do let me know, Brian if this is the right username). > Welcome aboard! > > Yep, that's the one. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Jan 7 04:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 07 Jan 2010 03:07:56 +0000 Subject: [Python-Dev] GIL required for _all_ Python calls? Message-ID: <4B45500C.8090905@mrabarnett.plus.com> Hi, I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? Or is it just a case of checking the source files for mentions of the GIL? The header file for PyList_New, for example, doesn't mention it! Thanks From john.arbash.meinel at gmail.com Thu Jan 7 04:25:48 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 06 Jan 2010 21:25:48 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B45543C.2090200@gmail.com> MRAB wrote: > Hi, > > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? Or is it just a case of checking the source > files for mentions of the GIL? The header file for PyList_New, for > example, doesn't mention it! > > Thanks Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may get concurrent updating of the value, and then the final value is wrong. (two threads do 5+1 getting 6, rather than 7, and when the decref, you end up at 4 rather than back at 5). AFAIK, the only things that don't require the GIL are macro functions, like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for example, will be increfing and setting the exception state, so certainly needs the GIL to be held. John =:-> From benjamin at python.org Thu Jan 7 04:32:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jan 2010 21:32:17 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45543C.2090200@gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> Message-ID: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> 2010/1/6 John Arbash Meinel : > Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may > get concurrent updating of the value, and then the final value is wrong. > (two threads do 5+1 getting 6, rather than 7, and when the decref, you > end up at 4 rather than back at 5). Correct. > > AFAIK, the only things that don't require the GIL are macro functions, > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > example, will be increfing and setting the exception state, so certainly > needs the GIL to be held. As a general rule, I would say, no Py* macros are safe without the gil either (the exception being Py_END_ALLOW_THREADS), since they mutate Python objects which must be protected. -- Regards, Benjamin From guido at python.org Thu Jan 7 05:29:03 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Jan 2010 20:29:03 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: On Wed, Jan 6, 2010 at 7:32 PM, Benjamin Peterson wrote: > 2010/1/6 John Arbash Meinel : > > AFAIK, the only things that don't require the GIL are macro functions, > > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > > example, will be increfing and setting the exception state, so certainly > > needs the GIL to be held. > > As a general rule, I would say, no Py* macros are safe without the gil > either (the exception being Py_END_ALLOW_THREADS), since they mutate > Python objects which must be protected. That's keeping it on the safe side, since there are some macros like PyString_AS_STRING() that are also safe, *if* you are owning at least one reference to the string object. At the same time, "no Py* macros" is not quite strong enough, since if you called PyString_AS_STRING() before releasing the GIL but you don't own a reference to the string object, the string might be deallocated behind your back by another thread. A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). -- --Guido van Rossum (python.org/~guido) From yoann.padioleau at facebook.com Thu Jan 7 08:24:42 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Wed, 6 Jan 2010 23:24:42 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt Message-ID: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a file called ast.txt. Where does this file come from ? Is it generated from Python.asdl ? From johan.gill at agama.tv Thu Jan 7 10:46:52 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 10:46:52 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. Message-ID: <4B45AD8C.5000405@agama.tv> Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before, so this is probably an already solved problem. Regards Johan Gill Agama Technologies From regebro at gmail.com Thu Jan 7 12:12:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 12:12:17 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45AD8C.5000405@agama.tv> References: <4B45AD8C.5000405@agama.tv> Message-ID: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I imagine that other companies have contributed before, so this is > probably an already solved problem. I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hodgestar+pythondev at gmail.com Thu Jan 7 12:28:01 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 7 Jan 2010 13:28:01 +0200 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon From swamy.sangamesh at gmail.com Thu Jan 7 11:46:59 2010 From: swamy.sangamesh at gmail.com (swamy sangamesh) Date: Thu, 7 Jan 2010 16:16:59 +0530 Subject: [Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9 Message-ID: Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have attached the full test case result. If i change the type c_int and c_short to c_unit and c_ushort of class "BITS(Structure)" in file test_bitfields.py then no failures are seen. Has anyone faced the similar issue or any help is appreciated. Thanks, Sangamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ctype-testcases Type: application/octet-stream Size: 22996 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 7 13:23:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 07 Jan 2010 22:23:11 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: <4B45D22F.40404@gmail.com> Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls we need to think >> about? I imagine that other companies have contributed before, so this is >> probably an already solved problem. > > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license without requesting additional permission from the copyright holder. For corporate contributions, I believe the contributor agreement needs to be signed by an authorised agent of the company - the place to check that would probably be psf at python.org (that's the email address for the PSF board). Assuming the subject line relates to the code that you would like to contribute though, that particular change is unlikely to happen - 2.6 is in maintenance mode and changing RLock from a Python implementation to the faster C one is solidly in new feature territory. Although a backport of the 3.2 C RLock implementation to 2.7 could be useful, I doubt that backporting code provided by an existing committer would be the subject of this query :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Thu Jan 7 14:11:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:11:27 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. Even if the contributed code as in this case is a method in an existing file? How does that work, how do they keep ownership of one method in the threading.py method? :-) > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) Ah. I probably misunderstood what the suggested contribution was. Maybe it was a separate file, which I didn't get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Thu Jan 7 14:15:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 07 Jan 2010 13:15:09 +0000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> Message-ID: <4B45DE5D.3030104@voidspace.org.uk> On 07/01/2010 13:11, Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > >> As Simon pointed out, while some organisations do work that way, the PSF >> isn't one of them. >> >> The PSF only requires that the code be contributed under a license that >> then allows us to turn around and redistribute it under a different open >> source license without requesting additional permission from the >> copyright holder. >> > Even if the contributed code as in this case is a method in an > existing file? How does that work, how do they keep ownership of one > method in the threading.py method? :-) > > When contributing code to Python all work remains copyright the original author. The combined work is copyright *everyone*. The PSF has a license to distribute it, which is all that is important. How you meaningfully exercise your ownership over chunks of code is left for the reader to determine... (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From stefan_ml at behnel.de Thu Jan 7 14:30:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:30:16 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: Guido van Rossum, 07.01.2010 05:29: > A better rule would be "you may access the memory buffer in a PyString > or PyUnicode object with the GIL released as long as you own a > reference to the string object." Everything else is out of bounds (or > not worth the bother). Is that a "yes" regarding the OP's original question about releasing the GIL during regexp searches? Stefan From regebro at gmail.com Thu Jan 7 14:36:42 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:36:42 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DE5D.3030104@voidspace.org.uk> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> <4B45DE5D.3030104@voidspace.org.uk> Message-ID: <319e029f1001070536t49f76899j111212b02c14f192@mail.gmail.com> On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Thu Jan 7 15:05:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:05:23 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB, 07.01.2010 04:07: > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. > or PyErr_SetString? Certainly not safe. > Is there an easy way to find out? Release it and fix any crashes? Note that this isn't a safe solution, though, as some GIL requiring code may be platform specific. So a better approach might be to extract any obviously problematic stuff from the existing code (such as any exception handling, explicit ref-counting or object creation), and *then* try to release the GIL. Stefan From solipsis at pitrou.net Thu Jan 7 16:38:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 15:38:31 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or macros you want to call with the GIL released, and assess whether it is safe to call them. As already pointed out, no reference count should be changed, and generally no mutable container should be accessed, except if that container is known not to be referenced anywhere else (that would be the case for e.g. a list that your function has created and is busy populating). I agree that releasing the GIL when doing non-trivial regex searches is a worthwhile research, so please don't give up immediately :-) Regards Antoine Pitrou. From olemis at gmail.com Thu Jan 7 19:24:59 2010 From: olemis at gmail.com (Olemis Lang) Date: Thu, 7 Jan 2010 13:24:59 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> Message-ID: <24ea26601001071024m2abd988fuc77f94845d57483a@mail.gmail.com> On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am strongly -1 for modifying the classes in ?traditional? unittest > module [2]_ (except that I am strongly +1 for the package structure > WITHOUT TOUCHING anything else ...) , and the more I think about it I > am more convinced ... but anyway, this not a big deal (and in the end > what I think is not that relevant either ... :o). So ... > IOW, if I had all the freedom to implement it, after the pkg structure I'd do something like : {{{ #!python class TestResult: # Everything just the same def _is_relevant_tb_level(self, tb): return '__unittest' in tb.tb_frame.f_globals class BetterTestResult(TestResult): # Further code ... maybe ;o) # def _is_relevant_tb_level(self, tb): # This or anything else you might want to do ;o) # globs = tb.tb_frame.f_globals is_relevant = '__name__' in globs and \ globs["__name__"].startswith("unittest") del globs return is_relevant }}} that's what inheritance is for ;o) ... but quite probably that's not gonna happen, just a comment . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Ubuntu sustituye GIMP por F-Spot - http://feedproxy.google.com/~r/simelo-es/~3/-g48D6T6Ojs/ubuntu-sustituye-gimp-por-f-spot.html From martin at v.loewis.de Thu Jan 7 21:24:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:24:32 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: <4B464300.2020204@v.loewis.de> >> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question about releasing the > GIL during regexp searches? No, because the regex engine may also operate on buffers that start moving around when you release the GIL. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:27:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:27:09 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B46439D.9030604@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in place - if their contents changes, regex results may be confusing. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:31:21 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Message-ID: <4B464499.4020305@v.loewis.de> > I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process a file > called ast.txt. Where does this file come from ? Is it generated from > Python.asdl ? astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. HTH, Martin From foom at fuhm.net Thu Jan 7 21:36:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 7 Jan 2010 15:36:37 -0500 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B46439D.9030604@v.loewis.de> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> On Jan 7, 2010, at 3:27 PM, Martin v. L?wis wrote: >> I've been wondering whether it's possible to release the GIL in the >> regex engine during matching. > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Even if they stay in place - > if their contents changes, regex results may be confusing. It seems probably worthwhile to optimize for the common case of using the regexp engine on an immutable object of type "str" or "bytes", and allow releasing the GIL in *that* case, even if you have to keep it for the general case. James From martin at v.loewis.de Thu Jan 7 21:45:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:45:42 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> Message-ID: <4B4647F6.2060309@v.loewis.de> >>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even if they stay in place - >> if their contents changes, regex results may be confusing. > > It seems probably worthwhile to optimize for the common case of using > the regexp engine on an immutable object of type "str" or "bytes", and > allow releasing the GIL in *that* case, even if you have to keep it for > the general case. Right. This problem was the one that I thought of first. Thinking about these things is fairly difficult (to me, at least), so I think I could only tell whether I would consider a patch thread-safe that released the GIL around matching under selected circumstances - if I had the patch available. I don't see any obvious reason (assuming Guido's list of conditions holds - i.e. you are holding references to everything you access). Regards, Martin From ncoghlan at gmail.com Thu Jan 7 21:48:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Jan 2010 06:48:05 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DECB.7070907@agama.tv> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <4B45DECB.7070907@agama.tv> Message-ID: <4B464885.40806@gmail.com> Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a contributor agreement was essential for this particular contribution would depend on how much new code was needed for the backport, but the bulk of the copyright on the C RLock code would remain with Antoine regardless. However, sorting through the legalities of the contributor agreement really is the best way to make sure every is squared away nice and neatly from a legal point of view. After all, even if I was a lawyer (which I'm not, I'm just a developer with an interest in licensing issues), I still wouldn't be *your* lawyer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Jan 7 21:43:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 20:43:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the representation isn't moved away until the buffer is released. Regards Antoine. From martin at v.loewis.de Thu Jan 7 21:59:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:59:29 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B464B31.7040406@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin From johan.gill at agama.tv Thu Jan 7 14:16:59 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 14:16:59 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <4B45DECB.7070907@agama.tv> On 01/07/2010 01:23 PM, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. For corporate contributions, I believe the contributor > agreement needs to be signed by an authorised agent of the company - the > place to check that would probably be psf at python.org (that's the email > address for the PSF board). > > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) > > Regards, > Nick. > > Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Regards Johan Gill Agama Technologies From yoann.padioleau at facebook.com Thu Jan 7 22:07:55 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Thu, 7 Jan 2010 13:07:55 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464499.4020305@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> Message-ID: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> On Jan 7, 2010, at 12:31 PM, Martin v. L?wis wrote: >> I would like to use astgen.py to generate python classes corresponding to the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take as an argument a .asdl file, but apparently it instead process a file >> called ast.txt. Where does this file come from ? Is it generated from >> Python.asdl ? > > astgen.py is not used to process asdl files; ast.txt lives right next to > astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a reflection mechanism things that were generated from ast.txt, then there could be a mismatch. Where does ast.txt comes from ? Shouldn't it be generated itself from Python.adsl ? So we would have Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py containing all the UnarySub, Expression, classes that represents a Python AST. > > HTH, > Martin From martin at v.loewis.de Thu Jan 7 22:11:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 07 Jan 2010 22:11:36 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <4B464E08.5020703@v.loewis.de> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the representation isn't moved away until the buffer is > released. In 2.7, we currently get the buffer with bf_getreadbuffer. In 3.x, we have /* Release the buffer immediately --- possibly dangerous but doing something else would require some re-factoring */ PyBuffer_Release(&view); Even if we do use the new API, and correctly, it still might be confusing if the contents of the buffer changes underneath. Regards, Martin From martin at v.loewis.de Thu Jan 7 22:16:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 22:16:12 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> Message-ID: <4B464F1C.7020404@v.loewis.de> >> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose as a reflection mechanism things that were generated from > ast.txt, then there could be a mismatch. Where does ast.txt comes > from ? Shouldn't it be generated itself from Python.adsl ? What you may not be aware of is that Tools/compiler (and the compiler package that it builds on) are both unused and unmaintained. If the package stops working correctly - tough luck. > So we would have > > Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py > containing all the UnarySub, Expression, classes that represents a > Python AST. No - what actually happens in Python 3.x is this: both the compiler package and Tools/compiler are removed. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 01:10:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 01:10:35 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM Message-ID: <201001080110.35800.victor.stinner@haypocalc.com> Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading an UTF-8 text file including a BOM: #7185 (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with the UTF-8-SIG encoding, but it's possible to do better. I propose to improve open() (TextIOWrapper) by using the BOM to choose the right encoding. I think that only files opened in read only mode should support this new feature. *Read* the BOM in a *write* only file would cause unexpected behaviours. Since my proposition changes the result TextIOWrapper.read()/readline() for files starting with a BOM, we might introduce an option to open() to enable the new behaviour. But is it really needed to keep the backward compatibility? I wrote a proof of concept attached to the issue #7651. My patch only changes the behaviour of TextIOWrapper for reading files starting with a BOM. It doesn't work yet if a seek() is used before the first read. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Jan 8 01:52:20 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 16:52:20 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner wrote: > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". > > See recent issues related to reading an UTF-8 text file including a BOM: #7185 > (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with > the UTF-8-SIG encoding, but it's possible to do better. > > I propose to improve open() (TextIOWrapper) by using the BOM to choose the > right encoding. I think that only files opened in read only mode should > support this new feature. *Read* the BOM in a *write* only file would cause > unexpected behaviours. > > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? > > I wrote a proof of concept attached to the issue #7651. My patch only changes > the behaviour of TextIOWrapper for reading files starting with a BOM. It > doesn't work yet if a seek() is used before the first read. > > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Fri Jan 8 03:23:08 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 02:23:08 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46970C.9010306@mrabarnett.plus.com> Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > Alternatively, have a universal UTF-8/16/32 encoding, ie one that expects UTF-8, with or without BOM, or UTF-16/32 with BOM. > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> >> See recent issues related to reading an UTF-8 text file including a BOM: #7185 >> (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with >> the UTF-8-SIG encoding, but it's possible to do better. >> >> I propose to improve open() (TextIOWrapper) by using the BOM to choose the >> right encoding. I think that only files opened in read only mode should >> support this new feature. *Read* the BOM in a *write* only file would cause >> unexpected behaviours. >> >> Since my proposition changes the result TextIOWrapper.read()/readline() for >> files starting with a BOM, we might introduce an option to open() to enable >> the new behaviour. But is it really needed to keep the backward compatibility? >> >> I wrote a proof of concept attached to the issue #7651. My patch only changes >> the behaviour of TextIOWrapper for reading files starting with a BOM. It >> doesn't work yet if a seek() is used before the first read. >> From nick.bastin at gmail.com Fri Jan 8 04:11:03 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 7 Jan 2010 22:11:03 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> Message-ID: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs to do this on FreeBSD as well (adding bsd to the list of platforms for which this is performed "solves" the problem, but I don't pretend to know enough about either distutils or freebsd to determine if this is the correct solution). -- Nick From glyph at twistedmatrix.com Fri Jan 8 04:34:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 7 Jan 2010 22:34:36 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? It *is* crazy, but unfortunately rather common. Wikipedia has a good description of the issues: . Basically, some Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as being UTF-8, so it's become a convention to do that. That's not good enough, so you need to guess the encoding as well to make sure, but if there is a BOM and you can otherwise verify that the file is probably UTF-8 encoded, you should discard it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Jan 8 05:17:28 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 07 Jan 2010 23:17:28 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solaris and automatically adds '.' to the library_dirs, and > I suspect it just needs to do this on FreeBSD as well (adding bsd to > the list of platforms for which this is performed "solves" the > problem, but I don't pretend to know enough about either distutils or > freebsd to determine if this is the correct solution). I wouldn't say it needed discussion on the SIG: just create a bug report, with the tentative patch you have worked out, and get it assigned to Tarek. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGsdQACgkQ+gerLs4ltQ5BMQCgtV8snMXH/6dDwgdN4sIJljLd koYAoKq6c0tKsRSrITHcygu4Od9FVzF5 =BJaE -----END PGP SIGNATURE----- From guido at python.org Fri Jan 8 05:21:04 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 20:21:04 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > > file starting with a BOM, read()/readline() returns also the BOM whereas the > > BOM should be "ignored". > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > > It *is* crazy, but unfortunately rather common. ?Wikipedia has a good > description of the issues: > . ?Basically, some > Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as > being UTF-8, so it's become a convention to do that. ?That's not good > enough, so you need to guess the encoding as well to make sure, but if there > is a BOM and you can otherwise verify that the file is probably UTF-8 > encoded, you should discard it. That doesn't make sense. If the file isn't UTF-8 you can't see the BOM, because the BOM itself is UTF-8-encoded. (And yes, I know this happens. Doesn't mean we need to auto-guess by default; there are lots of issues e.g. what should happen after seeking to offset 0?) -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Fri Jan 8 07:06:16 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 08 Jan 2010 15:06:16 +0900 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from the user for each instance. But it should do something sane when reading such files. I can't really see any harm in throwing it away, especially since use of ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated IIRC. From tseaver at palladion.com Fri Jan 8 07:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 01:12:12 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> >> BOM should be "ignored". >> >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. > > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) The BOM should not be seekeable if the file is opened with the proposed "guess encoding from BOM" mode: it isn't properly part of the stream at all in that case. A UTF-8 BOM is an absurditiy, but it exists *everywhere* in the wild: Python would do wll to make it as easy as possible to consume such files, as well as the non-insane versions (UTF-16 / UTF-32 BOMs). In the best of all possible worlds, I would just try opening the file so: f = open('/path/to/file', 'r', encoding="DWIFM") and any BOM present would set the encoding for the remainder of the stream.. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGzLsACgkQ+gerLs4ltQ5+cwCdGfycPdj6+cPfD23vH644SpHL sI0AoLGD7nfgMEJdJhBr90yjQQHfDgcJ =js+2 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Fri Jan 8 08:55:27 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 8 Jan 2010 02:55:27 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. It's just that the UTF-8 decoding of the BOM at the start of a file should be "". > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I think it's pretty clear that the BOM should still be skipped in that case ... From martin at v.loewis.de Fri Jan 8 10:05:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:05:17 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F54D.9090103@v.loewis.de> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I think what Glyph meant is this: if a file starts with the UTF-8 signature, assume it's UTF-8. Then validate the assumption against the rest of the file also, and then process it as UTF-8. If the rest clearly is not UTF-8, assume that the UTF-8 signature is bogus. I understood this proposal as a general processing guideline, not something the io library should do (but, say, a text editor). FWIW, I'm personally in favor of using the UTF-8 signature. If people consider them crazy talk, that may be because UTF-8 can't possibly have a byte order - hence I call it a signature, not the BOM. As a signature, I don't consider it crazy at all. There is a long tradition of having magic bytes in files (executable files, Postscript, PDF, ... - see /etc/magic). Having a magic byte sequence for plain text to denote the encoding is useful and helps reducing moji-bake. This is the reason it's used on Windows: notepad would normally assume that text is in the ANSI code page, and for compatibility, it can't stop doing that. So the UTF-8 signature gives them an exit strategy. Regards, Martin From martin at v.loewis.de Fri Jan 8 10:06:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:06:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <201001080110.35800.victor.stinner@haypocalc.com> <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4B46F59A.5060905@v.loewis.de> > But it should do something sane when reading such files. I can't > really see any harm in throwing it away, especially since use of > ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated > IIRC. And indeed it does, when you open the file in the utf-8-sig encoding. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 10:08:30 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 10:08:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46970C.9010306@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46970C.9010306@mrabarnett.plus.com> Message-ID: <201001081008.30904.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 03:23:08, MRAB a ?crit : > Guido van Rossum wrote: > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > > talk. And for the other two, perhaps it would make more sense to have > > a separate encoding-guessing function that takes a binary stream and > > returns a text stream wrapping it with the proper encoding? > > Alternatively, have a universal UTF-8/16/32 encoding, ie one that > expects UTF-8, > with or without BOM, or UTF-16/32 with BOM. Do you mean open(filename, encoding="BOM")? I suppose that "BOM" would be a magical value specific to read a text file (open(filename, "r")), not a real codec? Otherwise which encoding should be used for open(filename, "w", encoding="BOM")? -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Fri Jan 8 10:10:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:10:23 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F67F.60604@v.loewis.de> > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". It depends. If you use the utf-8-sig encoding, it *will* ignore the UTF-8 signature. > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? Absolutely. And there is no need to produce a new option, but instead use the existing options: define an encoding that auto-detects the encoding from the family of BOMs. Maybe you call it encoding="sniff". Regards, Martin From martin at v.loewis.de Fri Jan 8 10:11:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:11:51 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: <4B46F6D7.1050301@v.loewis.de> Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. I'm fairly skeptical that anybody on distutils SIG is interested in details of the Python build process... Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 11:27:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:27:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081127.44044.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : (...) > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I wrote a new version of my patch (version 3): * don't change the default behaviour: use open(filename, encoding="BOM") to check the BOM is there is any * fix for seek(0): always ignore the BOM * add an unit test: check that the right encoding is detect, but also the the BOM is ignored (especially after a seek(0)) BOM encoding doesn't work for writing into a file, so open(filename, "w", encoding="BOM") raises a ValueError. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:31:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:31:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081131.37492.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 01:52:20, Guido van Rossum a ?crit : > And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? I choosed to modify open()+TextIOWrapper instead of writing a new function because I would like to avoid an extra read operation (syscall) on the file. With my implementation, no specific read operation is needed to detect the BOM. The BOM is simply checked in the first _read_chunk(). -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:40:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:40:28 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Improve_open=28=29_to_support_reading?= =?iso-8859-1?q?_file_starting_with=09an_unicode_BOM?= In-Reply-To: <4B46F67F.60604@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> Message-ID: <201001081140.28124.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : > > Builtin open() function is unable to open an UTF-16/32 file starting with > > a BOM if the encoding is not specified (raise an unicode error). For an > > UTF-8 file starting with a BOM, read()/readline() returns also the BOM > > whereas the BOM should be "ignored". > > It depends. If you use the utf-8-sig encoding, it *will* ignore the > UTF-8 signature. Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to remove the BOM after the first read (much harder if you use a module like ConfigParser or csv). > > Since my proposition changes the result TextIOWrapper.read()/readline() > > for files starting with a BOM, we might introduce an option to open() to > > enable the new behaviour. But is it really needed to keep the backward > > compatibility? > > Absolutely. And there is no need to produce a new option, but instead > use the existing options: define an encoding that auto-detects the > encoding from the family of BOMs. Maybe you call it encoding="sniff". Good idea, I choosed open(filename, encoding="BOM"). -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri Jan 8 15:27:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:27:42 +0000 (UTC) Subject: [Python-Dev] GIL required for _all_ Python calls? References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: > > Even if we do use the new API, and correctly, it still might be > confusing if the contents of the buffer changes underneath. Well, no more confusing than when you compute a SHA1 hash or zlib- compress the buffer, is it? Regards Antoine From solipsis at pitrou.net Fri Jan 8 15:34:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:34:26 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any Well, I think if we implement this the default behaviour *should* be changed. It looks a bit senseless to have two different "auto-choose" options, one with encoding=None and one with encoding="BOM". Regards Antoine. From guido at python.org Fri Jan 8 16:48:44 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:48:44 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 6:27 AM, Antoine Pitrou wrote: > Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: >> >> Even if we do use the new API, and correctly, it still might be >> confusing if the contents of the buffer changes underneath. > > Well, no more confusing than when you compute a SHA1 hash or zlib- > compress the buffer, is it? That depends. Algorithms that make exactly one pass over the buffer will run fine (maybe producing a meaningless result). But the regex matcher may scan the buffer repeatedly (for backtracking purposes) and it would take a considerable analysis to prove that cannot mess up its internal data structures if the data underneath changes. (I give it a decent chance that it's fine, but since it was written without ever considering this possibility I'm not 100% sure.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:52:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:52:48 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 11:55 PM, Glyph Lefkowitz wrote: > I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. And I'm saying that it is, with as much certainty as we can ever guess the encoding of a file. > If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. ?It's just that the UTF-8 decoding of the BOM at the start of a file should be "". Sure, a Latin-1-encoded file could start with the same pattern that is a UTF-8-encoded BOM. But at that point, a UTF-16-encoded file is also valid Latin-1. The question was in the context of encoding-guessing; if we're guessing, a UTF-8-encoded BOM cannot signify anything else but UTF-8. (Ditto for UTF-16 and UTF-32 BOMs.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:54:15 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:54:15 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> >> I wrote a new version of my patch (version 3): >> >> ?* don't change the default behaviour: use open(filename, encoding="BOM") to >> check the BOM is there is any > > Well, I think if we implement this the default behaviour *should* be changed. > It looks a bit senseless to have two different "auto-choose" options, one with > encoding=None and one with encoding="BOM". Well there *are* two different auto options: use the environment variables (LANG etc.) or inspect the contents of the file. I think it would be useful to have ways to specify both. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:56:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:56:46 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 1:05 AM, "Martin v. L?wis" wrote: >>> It *is* crazy, but unfortunately rather common. ?Wikipedia has a good >>> description of the issues: >>> . ?Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. ?That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. > > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Sure. I said "crazy talk" only to stir up discussion. Which worked. :-) Also, I don't want Python's default behavior to change -- sniffing the encoding should be a separate option. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:59:45 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:59:45 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: > The BOM should not be seekeable if the file is opened with the proposed > "guess encoding from BOM" mode: ?it isn't properly part of the stream at > all in that case. This feels about right to me. There are still questions though: immediately after opening a file with a BOM, what should .tell() return? And regardless of that, .seek(0) should put the file in that same initial state. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Jan 8 17:03:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:03:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > > Well, I think if we implement this the default behaviour *should* be changed. > > It looks a bit senseless to have two different "auto-choose" options, one with > > encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Yes, perhaps. In the context of open() however I think it would be helpful to change the default. Moreover, reading the BOM is certainly much more reliable than our current guessing based on the locale or the "device encoding". Regards Antoine. From mal at egenix.com Fri Jan 8 17:25:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 17:25:22 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B475C72.1010207@egenix.com> Guido van Rossum wrote: > On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: >> Victor Stinner haypocalc.com> writes: >>> >>> I wrote a new version of my patch (version 3): >>> >>> * don't change the default behaviour: use open(filename, encoding="BOM") to >>> check the BOM is there is any >> >> Well, I think if we implement this the default behaviour *should* be changed. >> It looks a bit senseless to have two different "auto-choose" options, one with >> encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Shouldn't this encoding guessing be a separate function that you call on either a file or a seekable stream ? After all, detecting encodings is just as useful to have for non-file streams. You'd then avoid having to stuff everything into a single function call and also open up the door for more complex application specific guess work or defaults. The whole process would then have two steps: 1. guess encoding import codecs encoding = codecs.guess_file_encoding(filename) 2. open the file with the found encoding f = open(filename, encoding=encoding) For seekable streams f, you'd have: 1. guess encoding import codecs encoding = codecs.guess_stream_encoding(f) 2. wrap the stream with a reader for the found encoding reader_class = codecs.getreader(encoding) g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Jan 8 17:31:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:31:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver palladion.com> wrote: > > The BOM should not be seekeable if the file is opened with the proposed > > "guess encoding from BOM" mode: it isn't properly part of the stream at > > all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? tell() in the context of text I/O is specified to return an "opaque cookie". So whatever value it returns would probably be fine, as long as seeking to that value leaves the file in an acceptable state. Rewinding (seeking to 0) in the presence of a BOM is already reasonably supported by the TextIOWrapper object: >>> dec = codecs.getincrementaldecoder('utf-16')() >>> dec.decode(b'\xff\xfea\x00b\x00') 'ab' >>> dec.decode(b'\xff\xfea\x00b\x00') '\ufeffab' >>> >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' There are tests for this in test_io.py (test_encoded_writes, line 1929, and test_append_bom and test_seek_bom, line 2045). Regards Antoine. From python at mrabarnett.plus.com Fri Jan 8 17:47:18 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 16:47:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081127.44044.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B476196.4080409@mrabarnett.plus.com> Victor Stinner wrote: > Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : > (...) >> (And yes, I know this happens. Doesn't mean we need to auto-guess by >> default; there are lots of issues e.g. what should happen after >> seeking to offset 0?) > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any > * fix for seek(0): always ignore the BOM > * add an unit test: check that the right encoding is detect, but also the the > BOM is ignored (especially after a seek(0)) > > BOM encoding doesn't work for writing into a file, so open(filename, "w", > encoding="BOM") raises a ValueError. > I think it's similar to universal newline mode. You can tell it that you're reading UTF-something-encoded text (common forms only). The preference is UTF-8, but it could be UTF-8-sig (from Windows), or possibly UTF-16/32, which really need a BOM because there are multiple bytes per codepoint, so the bytes could be big-endian or little-endian. The BOM (or signature) tells you what the encoding is, defaulting to UTF-8 if there's none. If it subsequently raises a DecodeError, then so be it! Maybe there should also be a way of determining what encoding it decided it was, so that you can then write a new file in that same encoding. From status at bugs.python.org Fri Jan 8 18:07:13 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Jan 2010 18:07:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100108170714.014B078669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/01/10 - 01/08/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2544 open (+27) / 16937 closed (+15) / 19481 total (+42) Open issues with patches: 1017 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2509 (+27) pending 34 ( +0) Issues Created Or Reopened (43) _______________________________ Extended slicing with classic class behaves strangely 01/07/10 http://bugs.python.org/issue7532 reopened mark.dickinson patch optparse library documentation has an insignificant formatting i 01/01/10 CLOSED http://bugs.python.org/issue7618 created vazovsky patch imaplib shouldn't use cause DeprecationWarnings in 2.6 01/01/10 CLOSED http://bugs.python.org/issue7619 created djc Vim syntax highlight 01/02/10 http://bugs.python.org/issue7620 created july patch Test issue 01/02/10 CLOSED http://bugs.python.org/issue7621 created georg.brandl [patch] improve unicode methods: split() rsplit() and replace() 01/03/10 http://bugs.python.org/issue7622 created flox patch PropertyType missing in Lib/types.py 01/03/10 CLOSED http://bugs.python.org/issue7623 created wplappert isinstance(... ,collections.Callable) fails with oldstyle class 01/03/10 http://bugs.python.org/issue7624 created rgammans bytearray needs more tests for "b.some_method()[0] is not b" 01/03/10 http://bugs.python.org/issue7625 created flox patch Entity references without semicolon in HTMLParser 01/03/10 CLOSED http://bugs.python.org/issue7626 created stefan.schweizer mailbox.MH.remove() lock handling is broken 01/04/10 http://bugs.python.org/issue7627 created sraustein round() doesn't work correctly in 3.1.1 01/04/10 CLOSED http://bugs.python.org/issue7628 created bkovt Compiling with mingw32 gcc, content of variable "extra_postargs" 01/04/10 CLOSED http://bugs.python.org/issue7629 created popelkopp Strange behaviour of decimal.Decimal 01/04/10 CLOSED http://bugs.python.org/issue7630 created parmax undefined label: bltin-file-objects 01/04/10 CLOSED http://bugs.python.org/issue7631 created ezio.melotti dtoa.c: oversize b in quorem 01/04/10 http://bugs.python.org/issue7632 created skrah decimal.py: type conversion in context methods 01/04/10 http://bugs.python.org/issue7633 created skrah patch, easy next/previous links in documentation skip some sections 01/05/10 CLOSED http://bugs.python.org/issue7634 created gagenellina 19.6 xml.dom.pulldom doc: stub? 01/05/10 http://bugs.python.org/issue7635 created tjreedy Add a set update action to optparse 01/05/10 CLOSED http://bugs.python.org/issue7636 created hardkrash patch Improve 19.5. xml.dom.minidom doc 01/05/10 http://bugs.python.org/issue7637 created tjreedy Counterintuitive str.splitlines() inconsistency. 01/05/10 CLOSED http://bugs.python.org/issue7638 created vencabot_teppoo bdist_msi fails on files with long names 01/05/10 http://bugs.python.org/issue7639 created mmm77 buffered io seek() buggy 01/05/10 http://bugs.python.org/issue7640 created pakal patch Built-in Formatter accepts undocumented presentation type 01/06/10 CLOSED http://bugs.python.org/issue7641 created lrekucki [patch] Minor improvement in os.system doc 01/06/10 http://bugs.python.org/issue7642 created Val patch What is an ASCII linebreak? 01/06/10 http://bugs.python.org/issue7643 created flox bug in nntplib.body() method with possible fix 01/06/10 http://bugs.python.org/issue7644 created mdmullins easy test_distutils fails on Windows XP 01/06/10 http://bugs.python.org/issue7645 created austin987 test_telnetlib fails in Windows XP 01/06/10 http://bugs.python.org/issue7646 created austin987 Add statvfs flags to the posix module 01/06/10 http://bugs.python.org/issue7647 created ajax at redhat.com patch test_urllib2 fails on Windows if not run from C: 01/06/10 http://bugs.python.org/issue7648 created austin987 easy "u'%c' % char" broken for chars in range '\x80'-'\xFF' 01/07/10 http://bugs.python.org/issue7649 created ezio.melotti patch, easy test_uuid is invalid 01/07/10 http://bugs.python.org/issue7650 created austin987 Python3: guess text file charset using the BOM 01/07/10 http://bugs.python.org/issue7651 created haypo patch Merge C version of decimal into py3k. 01/07/10 http://bugs.python.org/issue7652 created mark.dickinson Fix description of the way the PythonPath Windows registry key w 01/07/10 CLOSED http://bugs.python.org/issue7653 created BoarGules patch, needs review [patch] Enable additional bytes and memoryview tests. 01/07/10 http://bugs.python.org/issue7654 created flox patch PEP 374 Title usage & script redirection typo fixes 01/07/10 CLOSED http://bugs.python.org/issue7655 created bab9e9 patch test_hashlib fails on some installations (specifically Neal's re 01/08/10 http://bugs.python.org/issue7656 created r.david.murray test_ctypes failure on AIX 5.3 01/08/10 http://bugs.python.org/issue7657 created mallayya patch OS X pythonw.c compile error with 10.4 or earlier deployment tar 01/08/10 http://bugs.python.org/issue7658 created ned.deily Problems with attribute assignment on object instances 01/08/10 http://bugs.python.org/issue7659 created pakal Issues Now Closed (40) ______________________ shutil fails when copying to NTFS in Linux 762 days http://bugs.python.org/issue1545 benjamin.peterson patch Test 751 days http://bugs.python.org/issue1619 georg.brandl Windows Registry issue with 2.5.2 AMD64 msi 645 days http://bugs.python.org/issue2539 loewis Extension module build fails for MinGW: missing vcvarsall.bat 618 days http://bugs.python.org/issue2698 Daniel26 optparse print_usage(),.. methods are not documented 542 days http://bugs.python.org/issue3340 ezio.melotti reference leaks in test_distutils 500 days http://bugs.python.org/issue3660 ezio.melotti patch _sha256 et al. encode to UTF-8 by default 20 days http://bugs.python.org/issue3745 lemburg 26backport Add Option to Bind to a Local IP Address in httplib.py 464 days http://bugs.python.org/issue3972 gregory.p.smith patch, easy no reference for optparse methods 388 days http://bugs.python.org/issue4635 ezio.melotti PyArg_Parse* should raise TypeError for float parsed with intege 339 days http://bugs.python.org/issue5080 mark.dickinson patch Don't use PyLong_SHIFT with _PyLong_AsScaledDouble() 282 days http://bugs.python.org/issue5576 mark.dickinson patch PyFrame_GetLineNumber 244 days http://bugs.python.org/issue5954 georg.brandl patch PyCode_NewEmpty 244 days http://bugs.python.org/issue5959 georg.brandl patch Add non-command help topics to help completion of cmd.Cmd 241 days http://bugs.python.org/issue5991 georg.brandl patch make error 231 days http://bugs.python.org/issue6067 georg.brandl no longer possible to hash arrays 229 days http://bugs.python.org/issue6071 benjamin.peterson patch zipfile: Invalid argument when opening zero-sized files 173 days http://bugs.python.org/issue6511 r.david.murray use different mechanism for pythonw on osx 127 days http://bugs.python.org/issue6834 ned.deily easy Py3k doc: "from __future__ import division" not necessary 32 days http://bugs.python.org/issue7432 ezio.melotti cPickle: stack underflow in load_pop() 31 days http://bugs.python.org/issue7455 pitrou patch crash in str.rfind() with an invalid start value 25 days http://bugs.python.org/issue7458 pitrou patch Implement fastsearch algorithm for rfind/rindex 26 days http://bugs.python.org/issue7462 ezio.melotti patch GZipFile.readline too slow 20 days http://bugs.python.org/issue7471 pitrou patch ssl module documentation: SSLSocket.unwrap description shown twi 5 days http://bugs.python.org/issue7592 georg.brandl Installation - 2 tests failed: test_cmd_line test_xmlrpc 7 days http://bugs.python.org/issue7601 ezio.melotti optparse library documentation has an insignificant formatting i 2 days http://bugs.python.org/issue7618 ezio.melotti patch imaplib shouldn't use cause DeprecationWarnings in 2.6 1 days http://bugs.python.org/issue7619 djc Test issue 0 days http://bugs.python.org/issue7621 ezio.melotti PropertyType missing in Lib/types.py 0 days http://bugs.python.org/issue7623 benjamin.peterson Entity references without semicolon in HTMLParser 2 days http://bugs.python.org/issue7626 r.david.murray round() doesn't work correctly in 3.1.1 0 days http://bugs.python.org/issue7628 benjamin.peterson Compiling with mingw32 gcc, content of variable "extra_postargs" 0 days http://bugs.python.org/issue7629 r.david.murray Strange behaviour of decimal.Decimal 0 days http://bugs.python.org/issue7630 mark.dickinson undefined label: bltin-file-objects 0 days http://bugs.python.org/issue7631 pitrou next/previous links in documentation skip some sections 0 days http://bugs.python.org/issue7634 georg.brandl Add a set update action to optparse 3 days http://bugs.python.org/issue7636 r.david.murray patch Counterintuitive str.splitlines() inconsistency. 0 days http://bugs.python.org/issue7638 flox Built-in Formatter accepts undocumented presentation type 0 days http://bugs.python.org/issue7641 eric.smith Fix description of the way the PythonPath Windows registry key w 0 days http://bugs.python.org/issue7653 r.david.murray patch, needs review PEP 374 Title usage & script redirection typo fixes 0 days http://bugs.python.org/issue7655 georg.brandl patch Top Issues Most Discussed (10) ______________________________ 22 [patch] improve unicode methods: split() rsplit() and replace() 5 days open http://bugs.python.org/issue7622 14 Test suite emits many DeprecationWarnings when -3 is enabled 91 days open http://bugs.python.org/issue7092 13 unicode_escape codec does not escape quotes 8 days open http://bugs.python.org/issue7615 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 0 days open http://bugs.python.org/issue7658 8 Merge C version of decimal into py3k. 1 days open http://bugs.python.org/issue7652 8 "u'%c' % char" broken for chars in range '\x80'-'\xFF' 2 days open http://bugs.python.org/issue7649 8 decimal.py: type conversion in context methods 4 days open http://bugs.python.org/issue7633 8 Patch for [ 735515 ] urllib2 should cache 301 redir 906 days open http://bugs.python.org/issue1755841 7 Test issue 0 days closed http://bugs.python.org/issue7621 7 Cannot use both read and readline method in same ZipExtFile obj 8 days open http://bugs.python.org/issue7610 From tseaver at palladion.com Fri Jan 8 22:09:54 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:09:54 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: >> The BOM should not be seekeable if the file is opened with the proposed >> "guess encoding from BOM" mode: it isn't properly part of the stream at >> all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? And regardless of that, .seek(0) should put the file in that > same initial state. I think the behavior should be something like: >>> f = open('/path/to/maybe-BOM-encoded-file', 'r', encoding='BOM') >>> f.tell() 0L >>> f.seek(-1) >>> f.tell() # count of unicode chars in decoded stream 45L >>> f.seek(0) >>> f.read(1) # read first unicode char decoded from stream. 'A' In other words, the BOM is not readable / seekable at all: it is invisible to the consumer of the decoded stream. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHnyIACgkQ+gerLs4ltQ6s3QCgznD+7FbUzfCbe5TS6OcoXjMg rdgAoJAMEXe2xwLCIwJaZ6XA6rVyTIAi =oXb3 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:19:10 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:19:10 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B475C72.1010207@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: > Shouldn't this encoding guessing be a separate function that you call > on either a file or a seekable stream ? > > After all, detecting encodings is just as useful to have for non-file > streams. Other stream sources typically have out-of-band ways to signal the encoding: only when reading from the filesystem do we pretty much *have* to guess, and in that case the BOM / signature is the best heuristic we have. Also, some non-file streams are not seekable, and so can't be guessed via a pre-pass. > You'd then avoid having to stuff everything into > a single function call and also open up the door for more complex > application specific guess work or defaults. > > The whole process would then have two steps: > > 1. guess encoding > > import codecs > encoding = codecs.guess_file_encoding(filename) Filename is not enough information: or do you mean that API to actually open the stream? > 2. open the file with the found encoding > > f = open(filename, encoding=encoding) > > For seekable streams f, you'd have: > > 1. guess encoding > > import codecs > encoding = codecs.guess_stream_encoding(f) > > 2. wrap the stream with a reader for the found encoding > > reader_class = codecs.getreader(encoding) > g = reader_class(f) > Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoU4ACgkQ+gerLs4ltQ5o3QCeLOJ7J91E+5f66vhgu1BUhYh4 9UgAnR2IeCd0BCsPez8ZilGNHJfhRn3Y =SoPb -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:14:59 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:14:59 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> It *is* crazy, but unfortunately rather common. Wikipedia has a good >>> description of the issues: >>> . Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. If the programmer opens the file using a "guess using the BOM" encoding, Python should *not* attempt to verify that the file is properly encoded: it should check for (and consume) any BOM, and then return a stream which uses the encoding inferred from the BOM. Any errors should be handled later, when characters are read, exactly as if the file had been opened with the same encoding guessed from the BOM. > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Agreed. Having that marker at the start of the file makes interop with other tools *much* easier. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoFMACgkQ+gerLs4ltQ73dACffwUfyh6Q9vUnKYf367QFjNcU RRMAoNuKCWEx7j+MSdTv+UjhAPynBc14 =uAX6 -----END PGP SIGNATURE----- From eric at trueblade.com Fri Jan 8 22:40:47 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 8 Jan 2010 16:40:47 -0500 (EST) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? >> >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. But what if the file were in (for example) a zip file? I think you definitely want to have access to this functionality outside of open(). Eric. From foom at fuhm.net Fri Jan 8 22:49:23 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 8 Jan 2010 16:49:23 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Jan 8, 2010, at 4:14 PM, Tres Seaver wrote: >> I understood this proposal as a general processing guideline, not >> something the io library should do (but, say, a text editor). >> >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly >> have >> a byte order - hence I call it a signature, not the BOM. As a >> signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote >> the >> encoding is useful and helps reducing moji-bake. This is the reason >> it's >> used on Windows: notepad would normally assume that text is in the >> ANSI >> code page, and for compatibility, it can't stop doing that. So the >> UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop > with > other tools *much* easier. Putting the BOM at the beginning of UTF-8 text files is not a good idea, it makes interop much *worse* on a unix system, not better. Without the BOM, most commands do the right thing with UTF-8 text. E.g. to concatenate two files: $ cat file-1 file-2 > file-3 With a BOM at the beginning of the file, it won't work right. Of course, you could modify "cat" (and every other stream processing command) to know how to consume and emit BOMs, and omit the extra one that would show up in the middle of the stream...but even that can't work; what about: $ (cat file-1; cat file-2) > file-3. Should the shell now know that when you run multiple commands, it should eat the BOM emitted from the second command? Basically, using a BOM in a utf-8 file is just not a good idea: it completely ruins interop with every standard unix tool. This is not to say that Python shouldn't have a way to read a file with a UTF-8 BOM: it just shouldn't encourage you to *write* such files. James From mal at egenix.com Fri Jan 8 22:51:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 22:51:26 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <4B47A8DE.1080401@egenix.com> Tres Seaver wrote: > M.-A. Lemburg wrote: > >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? > >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. Sure there are non-seekable file streams, but at least when using StringIO-type streams you don't have that restriction. An encoding detection function would provide more use in other cases as well, so instead of hiding away the functionality in the open() constructor, I'm suggesting to make expose it via the codecs module. Applications can then use it where necessary and also provide their own defaults, using other heuristics. >> You'd then avoid having to stuff everything into >> a single function call and also open up the door for more complex >> application specific guess work or defaults. > >> The whole process would then have two steps: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_file_encoding(filename) > > Filename is not enough information: or do you mean that API to actually > open the stream? Yes. The API would open the file, guess the encoding and then close it again. If you don't want that to happen, you could use the second API I mentioned below on the already open file. Note that this function could detect a lot more encodings with reasonably high probability than just BOM-prefixed ones, e.g. we could also add support to detect encoding declarations such as the ones we use in Python source files. >> 2. open the file with the found encoding > >> f = open(filename, encoding=encoding) > >> For seekable streams f, you'd have: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_stream_encoding(f) I forgot to mention: This API needs to position the file pointer to the start of the first data byte. >> 2. wrap the stream with a reader for the found encoding > >> reader_class = codecs.getreader(encoding) >> g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Fri Jan 8 22:59:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:59:04 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote: >>> Shouldn't this encoding guessing be a separate function that you call >>> on either a file or a seekable stream ? >>> >>> After all, detecting encodings is just as useful to have for non-file >>> streams. >> Other stream sources typically have out-of-band ways to signal the >> encoding: only when reading from the filesystem do we pretty much >> *have* to guess, and in that case the BOM / signature is the best >> heuristic we have. Also, some non-file streams are not seekable, and so >> can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). If the application expects a possibly-BOM-signature-marked file, but you pass it mismatched garbage: >>> f = open('some.zip', encoding='BOM") the error handling should be the same as if you passed any other mismatched encoding: >>> f = open('some.zip', encoding='UTF8') i.e., you discover the error when you try to read from the (non)encoded stream, not when you open it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHqpwACgkQ+gerLs4ltQ7uAACeKEc+WT4TASGcVl1Hfqe6L9La I6EAn1pJtngtLWPdothGbYB+zUabEvTW =TjBK -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Fri Jan 8 23:10:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 23:10:32 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: <201001082310.33029.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 22:40:47, Eric Smith a ?crit : > >> Shouldn't this encoding guessing be a separate function that you call > >> on either a file or a seekable stream ? > >> > >> After all, detecting encodings is just as useful to have for non-file > >> streams. > > > > Other stream sources typically have out-of-band ways to signal the > > encoding: only when reading from the filesystem do we pretty much > > *have* to guess, and in that case the BOM / signature is the best > > heuristic we have. Also, some non-file streams are not seekable, and so > > can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). FYI my patch (encoding="BOM") is implemented in TextIOWrapper, and TextIOWrapper takes a binary stream as input, not a filename. -- Victor Stinner http://www.haypocalc.com/ From yoann.padioleau at facebook.com Fri Jan 8 23:59:52 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Fri, 8 Jan 2010 14:59:52 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464F1C.7020404@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: On Jan 7, 2010, at 1:16 PM, Martin v. L?wis wrote: >>> astgen.py is not used to process asdl files; ast.txt lives right >>> next to astgen.py. Instead, the asdl file is processed by >>> Parser/asdl_c.py. >> >> Yes, I know that. That's why I asked about the relation between >> ast.txt and Python.adsl. If internally the parser uses the .adsl, but >> expose as a reflection mechanism things that were generated from >> ast.txt, then there could be a mismatch. Where does ast.txt comes >> from ? Shouldn't it be generated itself from Python.adsl ? > > What you may not be aware of is that Tools/compiler (and the > compiler package that it builds on) are both unused and unmaintained. I see. So if people want to analyze python code they have to use other tools (like rope?) ? or use reflection ? > > If the package stops working correctly - tough luck. > >> So we would have >> >> Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py >> containing all the UnarySub, Expression, classes that represents a >> Python AST. > > No - what actually happens in Python 3.x is this: both the compiler > package and Tools/compiler are removed. Ok. I will then create my own ast classes generator. Thanks. > > Regards, > Martin From g.brandl at gmx.net Sat Jan 9 00:10:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 09 Jan 2010 00:10:24 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: Am 08.01.2010 22:14, schrieb Tres Seaver: >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly have >> a byte order - hence I call it a signature, not the BOM. As a signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote the >> encoding is useful and helps reducing moji-bake. This is the reason it's >> used on Windows: notepad would normally assume that text is in the ANSI >> code page, and for compatibility, it can't stop doing that. So the UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop with > other tools *much* easier. Except if only 50% of the other tools support the signature. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jan 9 00:57:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 00:57:46 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: <4B47C67A.3020302@v.loewis.de> > I see. So if people want to analyze python code they have to use > other tools (like rope?) ? or use reflection ? Correct. One such tool might be the true Python compiler, along with the _ast module. Regards, Martin From victor.stinner at haypocalc.com Sat Jan 9 00:59:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 00:59:00 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM Message-ID: <201001090059.00848.victor.stinner@haypocalc.com> Hi, Thanks for all the answers! I will try to sum up all ideas here. (1) Change default open() behaviour or make it optional? Guido would like to add an option and keep open() unchanged. He wrote that checking for BOM and using system locale are too much different to be the same option (encoding=None). Antoine would like to check BOM by default, because both options (system locale vs checking for BOM) is the same thing. About Antoine choice (encoding=None): which file modes would check for a BOM? I would like to answer only the read only mode, but then open(filename, "r") and open(filename, "r+") would behave differently? => 1 point for Guido (encoding="BOM" is more explicit) Antoine choice has the advantage of directly support UTF-8+BOM, UTF-16 and UTF-32 for all applications and all modules using open(filename). => 1 point for Antoine (2) Check for a BOM while reading or detect it before? Everybody agree that checking BOM is an interesting option and should not be limited to open(). Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file name or a binary file-like object: it returns the encoding and seek to the file start or just after the BOM. I dislike this function because it requires extra file operations (open (optional), read() and seek()) and it doesn't work if the file is not seekable (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to avoid extra file operations. Note: I implemented the BOM check in TextIOWrapper; so it's already usable for any file-like object. (3) tell() and seek() on a text file starting with a BOM To be consistent with Antoine example: >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' TextIOWrapper: * tell() should return zero at file start, * seek(0) should go be to file start, * and the BOM should always be "ignored". I mean: with open("utf8bom.txt", encoding="BOM") as fp: assert fp.tell() == 0 text = fp.read() # no BOM here fp.seek(0) assert fp.read() == text -- About my patch: - BOM check is explicit: open(filebame, encoding="BOM") - tell() / seek(0) works as expected - BOM bytes are always skipped in read() / readlines() result Hum, I don't know if this email can be called a sum up ;-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 01:09:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 00:09:48 +0000 (UTC) Subject: [Python-Dev] Quick sum up about open() + BOM References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: Hello Victor, Victor Stinner haypocalc.com> writes: > > (1) Change default open() behaviour or make it optional? > [...] > > Antoine would like to check BOM by default, because both options (system > locale vs checking for BOM) is the same thing. To be clear, I am not saying it is the same thing. What I think is that it would be a mistake to use a mildly unreliable heuristic by default (the locale + device encoding heuristic) but refuse to trust a more reliable heuristic (the BOM-based detection algorithm). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Jan 9 01:14:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 00:14:39 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47CA6F.5000607@voidspace.org.uk> On 09/01/2010 00:09, Antoine Pitrou wrote: > Hello Victor, > > Victor Stinner haypocalc.com> writes: > >> (1) Change default open() behaviour or make it optional? >> >> > [...] > >> Antoine would like to check BOM by default, because both options (system >> locale vs checking for BOM) is the same thing. >> > To be clear, I am not saying it is the same thing. What I think is that it would > be a mistake to use a mildly unreliable heuristic by default (the locale + > device encoding heuristic) but refuse to trust a more reliable heuristic (the > BOM-based detection algorithm). > I concur. On Windows both UTF-8 and signature are very common, yet the platform default is the truly awful CP1252. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From floris.bruynooghe at gmail.com Sat Jan 9 01:26:05 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 9 Jan 2010 00:26:05 +0000 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B46F6D7.1050301@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> <4B46F6D7.1050301@v.loewis.de> Message-ID: <20100109002605.GB13536@laurie.devork> On Fri, Jan 08, 2010 at 10:11:51AM +0100, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > I think this problem probably needs to move over to distutils-sig, as > > it doesn't seem to be specific to the way that Python itself uses > > distutils. > > I'm fairly skeptical that anybody on distutils SIG is interested in > details of the Python build process... Uh, hum. Unfounded skepticism. ;-) But as said filing a bug sounds better in this case. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From v+python at g.nevcal.com Sat Jan 9 01:47:38 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 16:47:38 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47D22A.8070305@g.nevcal.com> On approximately 1/8/2010 3:59 PM, came the following characters from the keyboard of Victor Stinner: > Hi, > > Thanks for all the answers! I will try to sum up all ideas here. One concern I have with this implementation encoding="BOM" is that if there is no BOM it assumes UTF-8. That is probably a good assumption in some circumstances, but not in others. * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE encoded files include a BOM. It is only required that UTF-16 and UTF-32 (cases where the endianness is unspecified) contain a BOM. Hence, it might be that someone would expect a UTF-16LE (or any of the formats that don't require a BOM, rather than UTF-8), but be willing to accept any BOM-discriminated format. * Potentially, this could be expanded beyond the various Unicode encodings... one could envision that a program whose data files historically were in any particular national language locale, could want to be enhance to accept Unicode, and could declare that they will accept any BOM-discriminated format, but want to default, in the absence of a BOM, to the original national language locale that they historically accepted. That would provide a migration path for their old data files. So the point is, that it might be nice to have "BOM-otherEncodingForDefault" for each other encoding that Python supports. Not sure that is the right API, but I think it is expressive enough to handle the cases above. Whether the cases solve actual problems or not, I couldn't say, but they seem like reasonable cases. It would, of course, be nicest if OS metadata had been invented way back when, for all OSes, such that all text files were flagged with their encoding... then languages could just read the encoding and do the right thing! But we live in the real world, instead. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From python at mrabarnett.plus.com Sat Jan 9 02:12:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 09 Jan 2010 01:12:28 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B47D7FC.4070703@mrabarnett.plus.com> Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. > > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. > What about listing the possible encodings? It would try each in turn until it found one where the BOM matched or had no BOM: my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') or is that taking it too far? From martin at v.loewis.de Sat Jan 9 02:23:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 02:23:07 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47CA6F.5000607@voidspace.org.uk> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> Message-ID: <4B47DA7B.6050505@v.loewis.de> >>> Antoine would like to check BOM by default, because both options >>> (system locale vs checking for BOM) is the same thing. >>> >> To be clear, I am not saying it is the same thing. What I think is >> that it would be a mistake to use a mildly unreliable heuristic by >> default (the locale + device encoding heuristic) but refuse to >> trust a more reliable heuristic (the BOM-based detection >> algorithm). >> > > I concur. On Windows both UTF-8 and signature are very common, yet > the platform default is the truly awful CP1252. While I would support combining BOM detection in the case where a file is opened for reading and no encoding is specified, I see two problems: a) if a seek operations is performed before having looked at the BOM, no determination would have been made b) what encoding should it use on writing? Regards, Martin From v+python at g.nevcal.com Sat Jan 9 02:49:12 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 17:49:12 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <4B47E098.6030703@g.nevcal.com> On approximately 1/8/2010 5:12 PM, came the following characters from the keyboard of MRAB: > Glenn Linderman wrote: >> On approximately 1/8/2010 3:59 PM, came the following characters from >> the keyboard of Victor Stinner: >>> Hi, >>> >>> Thanks for all the answers! I will try to sum up all ideas here. >> >> One concern I have with this implementation encoding="BOM" is that if >> there is no BOM it assumes UTF-8. That is probably a good assumption >> in some circumstances, but not in others. >> >> * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE >> encoded files include a BOM. It is only required that UTF-16 and >> UTF-32 (cases where the endianness is unspecified) contain a BOM. >> Hence, it might be that someone would expect a UTF-16LE (or any of >> the formats that don't require a BOM, rather than UTF-8), but be >> willing to accept any BOM-discriminated format. >> >> * Potentially, this could be expanded beyond the various Unicode >> encodings... one could envision that a program whose data files >> historically were in any particular national language locale, could >> want to be enhance to accept Unicode, and could declare that they >> will accept any BOM-discriminated format, but want to default, in the >> absence of a BOM, to the original national language locale that they >> historically accepted. That would provide a migration path for their >> old data files. >> >> So the point is, that it might be nice to have >> "BOM-otherEncodingForDefault" for each other encoding that Python >> supports. Not sure that is the right API, but I think it is >> expressive enough to handle the cases above. Whether the cases solve >> actual problems or not, I couldn't say, but they seem like reasonable >> cases. >> >> It would, of course, be nicest if OS metadata had been invented way >> back when, for all OSes, such that all text files were flagged with >> their encoding... then languages could just read the encoding and do >> the right thing! But we live in the real world, instead. >> > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? That sounds very flexible -- but in net effect it would only make illegal a subset of the BOM-containing encodings (those not listed) without making legal any additional encodings other than the non-BOM encoding. Whether prohibiting a subset of BOM-containing encodings is a useful use case, I couldn't say... but my goal would be to included as many different file encodings on input as possible: without a BOM, that is exactly 1 (unless there are other heuristics), with a BOM, it is 1+all-BOM-containing encodings. Your scheme would permit numbers of encodings accepted to vary between 1 and 1+all-BOM-containing encodings. (I think everyone can agree there are 5 different byte sequences that can be called a Unicode BOM. The likelihood of them appearing in any other text encoding created by mankind depends on those other encodings -- but it is not impossible. It is truly up to the application to decide whether BOM detection could potentially conflict with files in some other encoding that would be acceptable to the application.) So I think it is taking it further than I can see value in, but I'm willing to be convinced otherwise. I see only a need for detecting BOM, and specifying a default encoding to be used if there is no BOM. Note that it might be nice to have a specification for using current encoding=None heuristic -- perhaps encoding="BOM-None" per my originally proposed syntax. But I'm still not saying that is the best syntax. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ncoghlan at gmail.com Sat Jan 9 04:07:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 09 Jan 2010 13:07:12 +1000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B476196.4080409@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B476196.4080409@mrabarnett.plus.com> Message-ID: <4B47F2E0.7090403@gmail.com> MRAB wrote: > Maybe there should also be a way of determining what encoding it decided > it was, so that you can then write a new file in that same encoding. I thought of that question as well - the f.encoding attribute on the opened file should be sufficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Sat Jan 9 06:48:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 06:48:36 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47E098.6030703@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> <4B47E098.6030703@g.nevcal.com> Message-ID: <319e029f1001082148h5cee2c0bn76f70a17766ca69e@mail.gmail.com> It seems to me that when opening a file, the following is the only flow that makes sense for the typical opening of a file flow: if encoding is not None: use encoding elif file has BOM: use BOM else: use system default And hence a encoding='BOM' isn't needed there. Although I'm trying to come up with usecases that doesn't work with this, I can't. :) BUT When writing things are not so easy though. Apparently some encodings require a BOM to be written, but others do not, but allow it, and some has no byte order mark. So there you have to be able to write the BOM, or not. And that's either a new parameter, because you can't use encoding='BOM' since you need to specify the encoding as well, or a new method. I would suggest a BOM parameter, and maybe a method as well. BOM=None|True|False Where "None" means a sane default behaviour, that is write a BOM if the encoding require it. "True" means write a BOM if the encoding *supports* it. "False" means Don't write a BOM even if the encoding requires it (because I know what I'm doing) if 'w' in mode: # But not 'r' or 'a' if BOM == True and encoding in (ENCODINGS THAT ALLOW BOM): write_bom = True elif BOM == False: write_bom = False elif BOM == None and encoding in (ENCODINGS THAT REQUIRE BOM): write_bom = True else: write_bom = False else: write_bom = False For reading this parameter could either be a noop, or possibly change the behavior somehow, if a usecase where that makes sense can be imagined. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From walter at livinglogic.de Sat Jan 9 11:51:57 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 11:51:57 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B485FCD.2040901@livinglogic.de> On 09.01.10 01:47, Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. This is doable with the currect API. Simply define a codec search function that handles all encoding names that start with "BOM-" and pass the "otherEncodingForDefault" part along to the codec. > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. Servus, Walter From walter at livinglogic.de Sat Jan 9 12:18:33 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 12:18:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081140.28124.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> Message-ID: <4B486609.2050804@livinglogic.de> Victor Stinner wrote: > Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : >>> Builtin open() function is unable to open an UTF-16/32 file starting with >>> a BOM if the encoding is not specified (raise an unicode error). For an >>> UTF-8 file starting with a BOM, read()/readline() returns also the BOM >>> whereas the BOM should be "ignored". >> It depends. If you use the utf-8-sig encoding, it *will* ignore the >> UTF-8 signature. > > Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and > UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to > remove the BOM after the first read (much harder if you use a module like > ConfigParser or csv). > >>> Since my proposition changes the result TextIOWrapper.read()/readline() >>> for files starting with a BOM, we might introduce an option to open() to >>> enable the new behaviour. But is it really needed to keep the backward >>> compatibility? >> Absolutely. And there is no need to produce a new option, but instead >> use the existing options: define an encoding that auto-detects the >> encoding from the family of BOMs. Maybe you call it encoding="sniff". > > Good idea, I choosed open(filename, encoding="BOM"). On the surface this looks like there's an encoding named "BOM", but looking at your patch I found that the check is still done in TextIOWrapper. IMHO the best approach would to the implement a *real* codec named "BOM" (or "sniff"). This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop. To see how something like this can be done, take a look at the UTF-16 codec, that switches to bigendian or littleendian mode depending on the first read/decode call. Servus, Walter From victor.stinner at haypocalc.com Sat Jan 9 13:37:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:37:06 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47DA7B.6050505@v.loewis.de> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> <4B47DA7B.6050505@v.loewis.de> Message-ID: <201001091337.06596.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:23:07, Martin v. L?wis a ?crit : > While I would support combining BOM detection in the case where a file > is opened for reading and no encoding is specified, I see two problems: > a) if a seek operations is performed before having looked at the BOM, > no determination would have been made TextIOWrapper doesn't support seek to an arbitrary byte. It uses "cookie" which is an opaque value. Reuse a cookie from another file or an old cookie is forbidden (but it doesn't raise an error). This is not specific to the BOM checking: the problem already exist for encodings using a BOM (eg. UTF-16). > b) what encoding should it use on writing? Don't change anything to writing. With Antoince choice: open('file.txt', 'w', encoding=None) continue to use the actual heuristic (os.device_encoding() or system locale). With Guido choice, encoding="BOM": it raises an error, because BOM check is not supported when writing into a file. How could the BOM be checked when creating a new (empty) file!? -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Sat Jan 9 13:45:58 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 09 Jan 2010 13:45:58 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B487A86.4020603@egenix.com> Victor Stinner wrote: > (2) Check for a BOM while reading or detect it before? > > Everybody agree that checking BOM is an interesting option and should not be > limited to open(). > > Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file > name or a binary file-like object: it returns the encoding and seek to the > file start or just after the BOM. > > I dislike this function because it requires extra file operations (open > (optional), read() and seek()) and it doesn't work if the file is not seekable > (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to > avoid extra file operations. > > Note: I implemented the BOM check in TextIOWrapper; so it's already usable for > any file-like object. Yes, but the implementation is limited to just BOM checking and thus only supports UTF-8-SIG, UTF-16 and UTF-32. With a codecs module function we could easily extend the encoding detection to more file types, e.g. XML files, Python source code files, etc. that use other mechanisms for defining the encoding. BTW: I haven't looked at your implementation, but what happens when your BOM check fails ? Will the implementation add the already read bytes back to a buffer ? This rollback action is the only reason for needing a seekable stream in codecs.guess_stream_encoding(). Another point to consider: AFAIK, we currently have a moratorium on changes to Python builtins. How does that match up with the proposed changes ? Using a new codec like Walter suggested would move the implementation into the stdlib for which doesn't the moratorium doesn't apply. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Sat Jan 9 13:54:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:54:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <201001091354.17239.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 01:47:38, vous avez ?crit : > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. If no BOM is found, it fallback to the current heuristic: os.device_encoding() or system local. > (...) Hence, it might be that someone would expect a UTF-16LE (or any of > the formats that don't require a BOM, rather than UTF-8), but be willing > to accept any BOM-discriminated format. > (...) declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted You mean "if there is a BOM, use it, otherwise fallback to a specific charset"? How could it be declared? Maybe: open("file.txt", check_bom=True, encoding="UTF16-LE") open("file.txt", check_bom=True, encoding="latin1") About falling back to UTF-8, it would be written: open("file.txt", check_bom=True, encoding="UTF-8") As explained before, check_bom=True is only accepted for read only file mode. Well, why not. This is a third choice for my point (1) :-) It's between Guido and Antoine choice, and I like it because we can fallback to UTF-8 instead of the dummy system locale: Windows users will be happy to be able to use UTF-8 :-) I prefer to fallback to a fixed encoding then depending on the system locale. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:34:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:34:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <201001091434.17521.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:12:28, MRAB a ?crit : > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? Yes, you're taking it foo far :-) Checking BOM is reliable, whereas *guessing* the charset only using the byte stream can only be an heuristic. Guess a charset is a complex problem, they are 3rd party library to do that, like the chardet project. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:38:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:38:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B486609.2050804@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <201001091438.43576.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : > > Good idea, I choosed open(filename, encoding="BOM"). > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Why not, this is another solution to the point (2) (Check for a BOM while reading or detect it before?). Which encoding would be used if there is not BOM? UTF-8 sounds like a good choice. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:50:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:50:28 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B487A86.4020603@egenix.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B487A86.4020603@egenix.com> Message-ID: <201001091450.28497.victor.stinner@haypocalc.com> Hi, Le samedi 09 janvier 2010 13:45:58, vous avez ?crit : > > Note: I implemented the BOM check in TextIOWrapper; so it's already > > usable for any file-like object. > > Yes, but the implementation is limited to just BOM checking > and thus only supports UTF-8-SIG, UTF-16 and UTF-32. Sure, but that's already better than no BOM check :-) It looks like many people would apprecite UTF-8-SIG detection, since this encoding is common on Windows. > BTW: I haven't looked at your implementation, but what happens > when your BOM check fails ? Will the implementation add the > already read bytes back to a buffer ? My implementation is done between buffer.read() and decoder.decode(data). If there is a BOM: set the encoding and remove the BOM bytes from the byte string. Otherwise, use another algorithm to choose the encoding and leave the byte string unchanged. It can be seen as a codec: it works like UTF-16 and UTF-32 codecs ;-) > AFAIK, we currently have a moratorium on changes to Python > builtins. How does that match up with the proposed changes ? Oh yes, I forgot the moratorium. In all solutions, some of them don't change the API. Eg. Antoine proposed to leave the API unchanged: open(file) => open(file) :-) I don't know if it's compatible with the moratorium or not. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 16:05:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 15:05:45 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: Walter D?rwald livinglogic.de> writes: > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Sorry but this is missing the point. The point here is to improve the open() function. I'm sure people who know about encodings are able to install the chardet library or even whip up their own BOM detection routine... Antoine. From benjamin at python.org Sat Jan 9 18:29:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 11:29:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 Message-ID: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> On behalf of the Python development team, I'm gleeful to announce the second alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From kmtracey at gmail.com Sat Jan 9 19:48:12 2010 From: kmtracey at gmail.com (Karen Tracey) Date: Sat, 9 Jan 2010 13:48:12 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce the > second > alpha release of Python 2.7. > > Well yay. Django's test suite (1242 tests) runs with just one failure on the 2.7 alpha 2 level, and that looks to be likely due to the improved string/float rounding so not really a problem, just a difference. That's down from 104 failures and 40 errors with 2.7 alpha 1. Note on the website page http://www.python.org/download/releases/2.7/ the "Change log for this release" link is still pointing to the alpha 1 changelog. Thanks, Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jan 9 19:51:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 12:51:11 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <1afaf6161001091051kb1ba08q9b96094af16c12fa@mail.gmail.com> 2010/1/9 Karen Tracey : > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson > wrote: >> >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> > > Well yay.? Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference.? That's > down from 104 failures and 40 errors with 2.7 alpha 1. Excellent! > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Thanks. I'll fix that. > -- Regards, Benjamin From skip at pobox.com Sat Jan 9 21:00:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:00:44 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 Message-ID: <19272.57452.972234.643008@montanaro.dyndns.org> How much of the Unladen Swallow cPickle speedups have been incorporated into 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the two versions I currently care about at work - we will skip 2.5 entirely). It appears some of their speedups may have already been merged to trunk, but I'm not sure how much. If a patch to merge this to 2.7 is already under consideration I won't look at it, but I interpreted Collin Winter's response to my query on the u-s mailing list that not everything has been done yet. Thx, Skip From martin at v.loewis.de Sat Jan 9 21:09:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 09 Jan 2010 21:09:27 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <4B48E277.9010408@v.loewis.de> Antoine Pitrou wrote: > Walter D?rwald livinglogic.de> writes: >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Sorry but this is missing the point. The point here is to improve the open() > function. I'm sure people who know about encodings are able to install the > chardet library or even whip up their own BOM detection routine... How does the requirement that it be implemented as a codec miss the point? FWIW, I agree with Walter that if it is provided through the encoding= argument, it should be a codec. If it is built into the open function (for whatever reason), it must be provided by some other parameter. I do see the point that it becomes available to end users only when released as part of Python. However, this *also* means that applications won't be using it for another three years or so, since they'll have to support older Python versions as well (unless it is integrated in the case where no encoding is specified). Regards, Martin From pjenvey at underboss.org Sat Jan 9 21:09:29 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 9 Jan 2010 12:09:29 -0800 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <19272.57452.972234.643008@montanaro.dyndns.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> On Jan 9, 2010, at 12:00 PM, skip at pobox.com wrote: > How much of the Unladen Swallow cPickle speedups have been incorporated into > 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the > two versions I currently care about at work - we will skip 2.5 entirely). > It appears some of their speedups may have already been merged to trunk, but > I'm not sure how much. If a patch to merge this to 2.7 is already under > consideration I won't look at it, but I interpreted Collin Winter's response > to my query on the u-s mailing list that not everything has been done yet. They've documented their upstream patches here: http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches -- Philip Jenvey From skip at pobox.com Sat Jan 9 21:21:20 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:21:20 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> Message-ID: <19272.58688.173011.412712@montanaro.dyndns.org> Philip> They've documented their upstream patches here: Philip> http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches Thanks. That will help immensely. Skip From solipsis at pitrou.net Sat Jan 9 21:28:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:28:12 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Sorry but this is missing the point. The point here is to improve the open() > > function. I'm sure people who know about encodings are able to install the > > chardet library or even whip up their own BOM detection routine... > > How does the requirement that it be implemented as a codec miss the > point? If we want it to be the default, it must be able to fallback on the current locale-based algorithm if no BOM is found. I don't think it would be easy for a codec to do that. > FWIW, I agree with Walter that if it is provided through the encoding= > argument, it should be a codec. If it is built into the open function > (for whatever reason), it must be provided by some other parameter. Why not simply encoding=None? The default value should provide the most useful behaviour possible. Forcing users to choose between two different autodetection strategies (encoding=None and another one) is a little insane IMO. Regards Antoine. From solipsis at pitrou.net Sat Jan 9 21:32:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:32:27 +0000 (UTC) Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > If a patch to merge this to 2.7 is already under > consideration I won't look at it, Why won't you look at it? :) Actually, if these patches are to be merged someone should certainly look at them, and do the (possibly) remaining work. http://bugs.python.org/issue5683 http://bugs.python.org/issue5671 Thank you Antoine. From skip at pobox.com Sat Jan 9 21:43:42 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:43:42 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <19272.60030.25319.269597@montanaro.dyndns.org> >>>>> "Antoine" == Antoine Pitrou writes: Antoine> pobox.com> writes: >> >> If a patch to merge this to 2.7 is already under >> consideration I won't look at it, Antoine> Why won't you look at it? :) I meant I wouldn't look at developing one. Skip From regebro at gmail.com Sat Jan 9 23:14:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 23:14:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Right. It seems like encoding=None is the right way to go there. encoding='BOM' would probably only work if 'BOM' isn't an encoding but a special tag, which is ugly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Sun Jan 10 00:25:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 23:25:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> Message-ID: <4B49105E.303@voidspace.org.uk> On 09/01/2010 22:14, Lennart Regebro wrote: > On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. >> > Right. It seems like encoding=None is the right way to go there. > encoding='BOM' would probably only work if 'BOM' isn't an encoding but > a special tag, which is ugly. > > I would rather see it as the default behavior for open without an encoding specified. I know Guido has expressed a preference against this so I won't continue to flog it. The current behavior however is that we have a 'guessing' algorithm based on the platform default. Currently if you open a text file in read mode that has a UTF-8 signature, but the platform default is something other than UTF-8, then we open the file using what is likely to be the incorrect encoding. Looking for the signature seems to be better behaviour in that case. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jan 10 00:40:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 10 Jan 2010 00:40:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <4B4913DF.7050801@v.loewis.de> >> How does the requirement that it be implemented as a codec miss the >> point? > > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Yes - however, Victor currently apparently *doesn't* want it to be the default, but wants the user to specify encoding="BOM". If so, it isn't the default, and it is easy to implement as a codec. >> FWIW, I agree with Walter that if it is provided through the encoding= >> argument, it should be a codec. If it is built into the open function >> (for whatever reason), it must be provided by some other parameter. > > Why not simply encoding=None? I don't mind. Please re-read Walter's message - it only said that *if* this is activated through encoding="BOM", *then* it must be a codec, and could be on PyPI. I don't think Walter was talking about the case "it is not activated through encoding='BOM'" *at all*. > The default value should provide the most useful > behaviour possible. Forcing users to choose between two different autodetection > strategies (encoding=None and another one) is a little insane IMO. That wouldn't disturb me much. There are a lot of things in that area that are a little insane, starting with Microsoft Windows :-) Regards, Martin From henning.vonbargen at arcor.de Sun Jan 10 12:10:02 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Sun, 10 Jan 2010 12:10:02 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: Message-ID: <4B49B58A.4040103@arcor.de> If Python should support BOM when reading text files, it should also be able to *write* such files. An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what? That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values: None: default (old) behaviour: don't handle BOM at all True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument. (*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False. Henning P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway... From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brett at python.org Sun Jan 10 19:51:26 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 10:51:26 -0800 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: Nick Coghlan thought I should forward this to python-dev so people are aware of this change and why it occurred (plus it indirectly informs Guido I finally finished the work). ---------- Forwarded message ---------- From: brett.cannon Date: Sat, Jan 9, 2010 at 18:56 Subject: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c To: python-checkins at python.org Author: brett.cannon Date: Sun Jan 10 03:56:19 2010 New Revision: 77402 Log: DeprecationWarning is now silent by default. This was originally suggested by Guido, discussed on the stdlib-sig mailing list, and given the OK by Guido directly to me. What this change essentially means is that Python has taken a policy of silencing warnings that are only of interest to developers by default. This should prevent users from seeing warnings which are triggered by an application being run against a new interpreter before the app developer has a chance to update their code. Closes issue #7319. Thanks to Antoine Pitrou, Ezio Melotti, and Brian Curtin for helping with the issue. Modified: python/trunk/Doc/library/warnings.rst python/trunk/Lib/test/test_ascii_formatd.py python/trunk/Lib/test/test_exceptions.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS python/trunk/Python/_warnings.c Modified: python/trunk/Doc/library/warnings.rst ============================================================================== --- python/trunk/Doc/library/warnings.rst (original) +++ python/trunk/Doc/library/warnings.rst Sun Jan 10 03:56:19 2010 @@ -59,7 +59,7 @@ | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,14 +151,6 @@ :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - .. _warning-suppress: @@ -226,6 +221,37 @@ entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions Modified: python/trunk/Lib/test/test_ascii_formatd.py ============================================================================== --- python/trunk/Lib/test/test_ascii_formatd.py (original) +++ python/trunk/Lib/test/test_ascii_formatd.py Sun Jan 10 03:56:19 2010 @@ -4,6 +4,7 @@ import unittest from test_support import check_warnings, run_unittest, cpython_only +import warnings class FormatDeprecationTests(unittest.TestCase): @@ -17,6 +18,7 @@ buf = create_string_buffer(' ' * 100) with check_warnings() as w: + warnings.simplefilter('default') PyOS_ascii_formatd(byref(buf), sizeof(buf), '%+.10f', c_double(10.0)) self.assertEqual(buf.value, '+10.0000000000') Modified: python/trunk/Lib/test/test_exceptions.py ============================================================================== --- python/trunk/Lib/test/test_exceptions.py (original) +++ python/trunk/Lib/test/test_exceptions.py Sun Jan 10 03:56:19 2010 @@ -309,6 +309,7 @@ # BaseException.__init__ triggers a deprecation warning. exc = BaseException("foo") with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('default') self.assertEquals(exc.message, "foo") self.assertEquals(len(w), 1) self.assertEquals(w[0].category, DeprecationWarning) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Sun Jan 10 03:56:19 2010 @@ -383,8 +383,8 @@ # Module initialization _processoptions(sys.warnoptions) if not _warnings_defaults: - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) + for cls in (DeprecationWarning, PendingDeprecationWarning, ImportWarning): + simplefilter("ignore", category=cls, append=True) bytes_warning = sys.flags.bytes_warning if bytes_warning > 1: bytes_action = "error" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Jan 10 03:56:19 2010 @@ -12,6 +12,8 @@ Core and Builtins ----------------- +- Issue #7319: Silence DeprecationWarning by default. + - Issue #2335: Backport set literals syntax from Python 3.x. Library Modified: python/trunk/Python/_warnings.c ============================================================================== --- python/trunk/Python/_warnings.c (original) +++ python/trunk/Python/_warnings.c Sun Jan 10 03:56:19 2010 @@ -85,10 +85,10 @@ default_action = get_warnings_attr("defaultaction"); if (default_action == NULL) { - if (PyErr_Occurred()) { - return NULL; - } - return _default_action; + if (PyErr_Occurred()) { + return NULL; + } + return _default_action; } Py_DECREF(_default_action); @@ -202,12 +202,12 @@ mod_str = PyString_AsString(filename); if (mod_str == NULL) - return NULL; + return NULL; len = PyString_Size(filename); if (len < 0) return NULL; if (len >= 3 && - strncmp(mod_str + (len - 3), ".py", 3) == 0) { + strncmp(mod_str + (len - 3), ".py", 3) == 0) { module = PyString_FromStringAndSize(mod_str, len-3); } else { @@ -251,7 +251,7 @@ name = PyObject_GetAttrString(category, "__name__"); if (name == NULL) /* XXX Can an object lack a '__name__' attribute? */ - return; + return; f_stderr = PySys_GetObject("stderr"); if (f_stderr == NULL) { @@ -341,7 +341,7 @@ rc = already_warned(registry, key, 0); if (rc == -1) goto cleanup; - else if (rc == 1) + else if (rc == 1) goto return_none; /* Else this warning hasn't been generated before. */ } @@ -492,9 +492,9 @@ /* Setup filename. */ *filename = PyDict_GetItemString(globals, "__file__"); if (*filename != NULL) { - Py_ssize_t len = PyString_Size(*filename); + Py_ssize_t len = PyString_Size(*filename); const char *file_str = PyString_AsString(*filename); - if (file_str == NULL || (len < 0 && PyErr_Occurred())) + if (file_str == NULL || (len < 0 && PyErr_Occurred())) goto handle_error; /* if filename.lower().endswith((".pyc", ".pyo")): */ @@ -506,10 +506,10 @@ tolower(file_str[len-1]) == 'o')) { *filename = PyString_FromStringAndSize(file_str, len-1); - if (*filename == NULL) - goto handle_error; - } - else + if (*filename == NULL) + goto handle_error; + } + else Py_INCREF(*filename); } else { @@ -536,8 +536,8 @@ else { /* embedded interpreters don't have sys.argv, see bug #839151 */ *filename = PyString_FromString("__main__"); - if (*filename == NULL) - goto handle_error; + if (*filename == NULL) + goto handle_error; } } if (*filename == NULL) { @@ -839,26 +839,29 @@ static PyObject * init_filters(void) { - PyObject *filters = PyList_New(3); + PyObject *filters = PyList_New(4); const char *bytes_action; if (filters == NULL) return NULL; PyList_SET_ITEM(filters, 0, + create_filter(PyExc_DeprecationWarning, "ignore")); + PyList_SET_ITEM(filters, 1, create_filter(PyExc_PendingDeprecationWarning, "ignore")); - PyList_SET_ITEM(filters, 1, create_filter(PyExc_ImportWarning, "ignore")); + PyList_SET_ITEM(filters, 2, create_filter(PyExc_ImportWarning, "ignore")); if (Py_BytesWarningFlag > 1) bytes_action = "error"; else if (Py_BytesWarningFlag) bytes_action = "default"; else bytes_action = "ignore"; - PyList_SET_ITEM(filters, 2, create_filter(PyExc_BytesWarning, + PyList_SET_ITEM(filters, 3, create_filter(PyExc_BytesWarning, bytes_action)); if (PyList_GET_ITEM(filters, 0) == NULL || PyList_GET_ITEM(filters, 1) == NULL || - PyList_GET_ITEM(filters, 2) == NULL) { + PyList_GET_ITEM(filters, 2) == NULL || + PyList_GET_ITEM(filters, 3) == NULL) { Py_DECREF(filters); return NULL; } _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sun Jan 10 20:22:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 10 Jan 2010 20:22:10 +0100 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: <201001102022.10259.victor.stinner@haypocalc.com> Hi, Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : > Nick Coghlan thought I should forward this to python-dev so people are > aware of this change and why it occurred (plus it indirectly informs Guido > I finally finished the work). Thanks to copy this mail to the python-dev mailing list. What is the recommanded way to get the previous behaviour (print deprecation warnings once)? Is there a command line option? Is it "-W default"? -- Victor Stinner http://www.haypocalc.com/ From benjamin at python.org Sun Jan 10 20:23:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 13:23:54 -0600 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <201001102022.10259.victor.stinner@haypocalc.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> <201001102022.10259.victor.stinner@haypocalc.com> Message-ID: <1afaf6161001101123g366d80dbjeaa80f1a632605e2@mail.gmail.com> 2010/1/10 Victor Stinner : > Hi, > > Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : >> Nick Coghlan thought I should forward this to python-dev so people are >> ?aware of this change and why it occurred (plus it indirectly informs Guido >> ?I finally finished the work). > > Thanks to copy this mail to the python-dev mailing list. > > What is the recommanded way to get the previous behaviour (print deprecation > warnings once)? Is there a command line option? Is it "-W default"? That commit conveniently adds documentation answering that question. :) -- Regards, Benjamin From nas at arctrix.com Sun Jan 10 20:30:09 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:30:09 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce > the second alpha release of Python 2.7. Thanks to everyone who contributed. > Python 2.7 is scheduled to be the last major version in the 2.x > series. Has this really been decided already? Maybe I missed it. In my opinion, it does Python's reputation harm to make such a statement. Conservative users (probably the vast majority of Python users) don't like to hear that software they are considering using is nearing the end of its life. What does it gain us to announce that the 2.x branch is dead aside from bugfixes? I propose that the 2.x branch be treated like 2.x.y branches: as long as there is sufficient volunteer labour, it should continue to live. In order to avoid wasted development effort, it would be prudent to announce that unless a 2.8 release manager steps up, whatever is committed to the 2.x branch after the 2.7 release may never get released. Said another way, it's okay for the Python developers to decide to abandon 2.x and put their efforts into 3.x. It's not okay for them to prevent others from continuing to work on 2.x or to somehow make 2.x look worse so 3.x looks better. Python 3 needs to stand on its own terms and I'm confident it can. Best regards, Neil From brett at python.org Sun Jan 10 21:09:08 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:09:08 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sun, Jan 10, 2010 at 11:30, Neil Schemenauer wrote: > Benjamin Peterson wrote: > > On behalf of the Python development team, I'm gleeful to announce > > the second alpha release of Python 2.7. > > Thanks to everyone who contributed. > > > Python 2.7 is scheduled to be the last major version in the 2.x > > series. > > Has this really been decided already? Maybe I missed it. More or less. It was first discussed at the language summit last year and has come up here a couple of times. If needed we can make it official in terms of lifetime of 2.7, etc. at the language summit this year. > In my > opinion, it does Python's reputation harm to make such a statement. > Conservative users (probably the vast majority of Python users) > don't like to hear that software they are considering using is > nearing the end of its life. What does it gain us to announce that > the 2.x branch is dead aside from bugfixes? > > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. > > Said another way, it's okay for the Python developers to decide to > abandon 2.x and put their efforts into 3.x. It's not okay for them > to prevent others from continuing to work on 2.x or to somehow make > 2.x look worse so 3.x looks better. Python 3 needs to stand on its > own terms and I'm confident it can. > I don't think ending the 2.x series at 2.7 makes it look bad compared to 3.2; it's simply the end of a development line like any other software project. I suspect 2.7 will have a protracted bugfix window because so much code runs on 2.x exclusively at the moment. And if core developers want to continue to backport fixes past two years from release they can (or however long we decide to officially support 2.7). No one is saying people still can't work on the code, just that python-dev as an entity is not going to focus its effort on the 2.x series anymore and people should not rely upon us to continue to focus new development efforts in that series. If there are core developers who want to continue to do bugfix releases then that's fine and I am happy to flag patches as needing backports and let other's do the work after the standard two year maintenance cycle, but I know I do not want to be held accountable as a core developer to keep the 2.x going indefinitely. Maintaining four branches is enough of a reason in my book to not keep the 2.x series going. If there really is an outcry on this we can re-visit the issue, but as of right now we need to move forward at some point and 2.7 seems like that good point. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 10 21:23:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Jan 2010 06:23:27 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A373F.9050601@gmail.com> Neil Schemenauer wrote: > In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. The announcement is precisely to avoid the situation where people commit new features to the 2.x main line of development (after the 2.7 maintenance branch is created) in the expectation that they will be released as part of a hypothetical 2.8 release. Whether that info needs to be in each and every 2.7 announcement... probably not. It isn't really info for users of Python, just for developers of Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jan 10 21:25:29 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:25:29 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit Message-ID: Michael has given me the hg transition/stdlib time slot at the language summit this year. In regards to that I plan to lead a discussion on: * where we are at w/ the Hg transition (Dirkjan should be there and I did a blog post on this topic recently: http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) * argparse (PEP 389) * brief mention on still wanting to break out the stdlib from CPython * an official policy on extension modules? (i.e. must have a pure Python implementation which can mean a ctypes implementation unless given an explicit waiver) That's everything from a stdlib perspective. I have half-baked ideas, but nothing concrete enough to bring up unless people really want to discuss stuff like how to potentially standardize module deprecation warnings, etc. In terms of me personally, I do plan to bring up at some point during the summit these points which don't squarely fit during my time slot: * an official unofficial policy on how new proposed features should come to us (i.e. working code to python-ideas with a shell of a PEP that includes abstract and motivation -> hashed out PEP to python-dev -> pronouncement) * any changes needed to the issue tracker to help with the workflow? (stage field seems like a failed experiment and we now have several effective triage people who can help w/ guiding changes) If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Sun Jan 10 22:52:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 10 Jan 2010 22:52:00 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) Unfortunately my flight doesn't land until about the time the Mercurial slot ends, so while I'll be there later on that afternoon for sprinting or questions or anything, I won't be at the actual Mercurial talk at the summit. Cheers, Dirkjan From fuzzyman at voidspace.org.uk Sun Jan 10 23:27:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 10 Jan 2010 22:27:58 +0000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4A546E.8010808@voidspace.org.uk> On 10/01/2010 21:52, Dirkjan Ochtman wrote: > On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > >> Michael has given me the hg transition/stdlib time slot at the language >> summit this year. In regards to that I plan to lead a discussion on: >> * where we are at w/ the Hg transition (Dirkjan should be there and I did a >> blog post on this topic recently: >> http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) >> > Unfortunately my flight doesn't land until about the time the > Mercurial slot ends, so while I'll be there later on that afternoon > for sprinting or questions or anything, I won't be at the actual > Mercurial talk at the summit. > > We will probably be in 'open discussion' when you arrive so it will still be good to hear from you on the topic and there maybe questions that can't be answered until you arrive... :-) Michael > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Mon Jan 11 00:02:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:02:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A5C69.7090007@v.loewis.de> > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. I think it's more difficult than that. It also takes developer effort to *commit* to the trunk. So if the trunk continued to live, would it still be ok if I closed a 2.x feature request as "won't fix", or only committed the new feature to the 3.x branch? As for old branches: they *don't* live in the way you claim (i.e. remain open with changes potentially just not released). Instead, at some point, they are frozen to bug-fix only, then to security-fix only, and then to no change at all. Regards, Martin From martin at v.loewis.de Mon Jan 11 00:07:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:07:58 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A373F.9050601@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> Message-ID: <4B4A5DCE.3070909@v.loewis.de> > The announcement is precisely to avoid the situation where people commit > new features to the 2.x main line of development (after the 2.7 > maintenance branch is created) in the expectation that they will be > released as part of a hypothetical 2.8 release. > > Whether that info needs to be in each and every 2.7 announcement... > probably not. It isn't really info for users of Python, just for > developers of Python. I think the announcement is indeed also for the users, and I would prefer it to stay in the release announcements, so that people will know for sure (and speak up) before the 2.7 release happens. As for decisions: I don't think there was an official BDFL pronouncent, but I recall Guido posting a message close to that, proposing that 2.7 will be a release that will see bug fix releases for an indefinite period of time (where indefinite != infinite). This was shortly after him proposing that perhaps we shouldn't make a 2.7 release at all, and stop at 2.6. As for such a decision giving a bad light on Python: I don't think that will be the case. Instead, I hear many users surprised for how long we have been maintaining to parallel versions - "that must have taken a lot of effort". So everybody will likely understand that enough is enough. Regards, Martin From benjamin at python.org Mon Jan 11 01:07:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 18:07:35 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Consider this program: class Descr(object): def __init__(self, name): self.name = name def __set__(self, instance, what): instance.__dict__[self.name] = what class X(object): attr = Descr("attr") x = X() print(x.attr) x.attr = 42 print(x.attr) It gives in output: <__main__.Descr object at 0x7fe1c9b28150> 42 The documentation [1] says that Descr is a data descriptor because it defines the __set__ method. It also states that data descriptors always override the value in the instance dictionary. So, the second line should also be the descriptor object according to the documentation. My question is: Is this a doc bug or a implementation bug? If the former, it will be the description of a data descriptor much less consistent, since it will require that a __get__ method be present, too. If the latter, the fix may break some programs relying on the ability to "cache" a value in the instance dictionary. [1] http://docs.python.org/reference/datamodel#invoking-descriptors -- Regards, Benjamin From amauryfa at gmail.com Mon Jan 11 01:51:09 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Jan 2010 01:51:09 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: Hi, 2010/1/11 Benjamin Peterson : > Consider this program: > > class Descr(object): > ? ?def __init__(self, name): > ? ? ? ?self.name = name > ? ?def __set__(self, instance, what): > ? ? ? ?instance.__dict__[self.name] = what > > class X(object): > ? ?attr = Descr("attr") > > x = X() > print(x.attr) > x.attr = 42 > print(x.attr) > > It gives in output: > > <__main__.Descr object at 0x7fe1c9b28150> > 42 > > The documentation [1] says that Descr is a data descriptor because it > defines the __set__ method. It also states that data descriptors > always override the value in the instance dictionary. So, the second > line should also be the descriptor object according to the > documentation. > > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors Quoting the documentation: """Normally, data descriptors define both __get__() and __set__(), while non-data descriptors have just the __get__() method. """ Your example is neither a data descriptor nor a non-data descriptor... The thing that worries me a bit is the "x.attr" returning the Descr object. Descriptors should remain at the class level, and instance should only see values. I'd prefer an AttributeError in this case. -- Amaury Forgeot d'Arc From benjamin at python.org Mon Jan 11 02:00:32 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 19:00:32 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <1afaf6161001101700u21006708l2ef62bfa257e4ce7@mail.gmail.com> 2010/1/10 Amaury Forgeot d'Arc : > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... See the footnote: http://docs.python.org/reference/datamodel#id7 > > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. Far too late for that, I'm afraid. -- Regards, Benjamin From nas at arctrix.com Mon Jan 11 02:44:57 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:44:57 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <20100111014457.GA5289@arctrix.com> On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > I don't think ending the 2.x series at 2.7 makes it look bad > compared to 3.2; it's simply the end of a development line like > any other software project. I suspect 2.7 will have a protracted > bugfix window because so much code runs on 2.x exclusively at the > moment. I would guess over 99% of all Python code written doesn't run on Python 3. Given that, I think it is premature to close the door on new major versions of Python 2.x. Also, we as a project should be careful not to present the image that Python 2.x will not be supported in the future. > If there really is an outcry on this we can re-visit the issue, > but as of right now we need to move forward at some point and 2.7 > seems like that good point. I think that's bad PR. If I had a successful product, I would not announce its end of life just to see how many customers scream and then decide if I should devote more resources to continue maintaining it. IMHO, the release notes should say something like: After the Python 2.7 release, the focus of Python development will be on Python 3. There will continue to be maintainance releases of Python 2.x. trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil From tjreedy at udel.edu Mon Jan 11 03:26:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Jan 2010 21:26:34 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On 1/10/2010 8:44 PM, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: >> I don't think ending the 2.x series at 2.7 makes it look bad >> compared to 3.2; it's simply the end of a development line like >> any other software project. I suspect 2.7 will have a protracted >> bugfix window because so much code runs on 2.x exclusively at the >> moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. If the removal of old features had been done in the 2.x series, as once planned (Guido originally proposed removing the old meaning of int / int in 2.5) the same more or less would be true of 2.7. It is past time for other old and now duplicated features to be removed also. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > >> If there really is an outcry on this we can re-visit the issue, >> but as of right now we need to move forward at some point and 2.7 >> seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. Python is not being ended, but upgraded (with bloating duplications removed). Think of 3.1 as 2.8 with a new name. tjr From lrekucki at gmail.com Mon Jan 11 04:26:48 2010 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Mon, 11 Jan 2010 04:26:48 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: > Date: Mon, 11 Jan 2010 01:51:09 +0100 > From: "Amaury Forgeot d'Arc" > To: Benjamin Peterson > Cc: Python Dev > Subject: Re: [Python-Dev] Data descriptor doc/implementation > ? ? ? ?inconsistency > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > 2010/1/11 Benjamin Peterson : >> Consider this program: >> >> class Descr(object): >> ? ?def __init__(self, name): >> ? ? ? ?self.name = name >> ? ?def __set__(self, instance, what): >> ? ? ? ?instance.__dict__[self.name] = what >> >> class X(object): >> ? ?attr = Descr("attr") >> >> x = X() >> print(x.attr) >> x.attr = 42 >> print(x.attr) >> >> It gives in output: >> >> <__main__.Descr object at 0x7fe1c9b28150> >> 42 >> >> The documentation [1] says that Descr is a data descriptor because it >> defines the __set__ method. It also states that data descriptors >> always override the value in the instance dictionary. So, the second >> line should also be the descriptor object according to the >> documentation. >> >> My question is: Is this a doc bug or a implementation bug? If the >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors > > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... Actually, there is this footnote [1]: """ A descriptor can define any combination of __get__(), __set__() and __delete__(). If it does not define __get__(), then accessing the attribute even on an instance will return the descriptor object itself. If the descriptor defines __set__() and/or __delete__(), it is a data descriptor; if it defines neither, it is a non-data descriptor. """ Which would mean Descr is actually a data descriptor without a __get__(), so x.attr should always return the descriptor object itself (at least in the docs). > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. > > -- > Amaury Forgeot d'Arc [1] http://docs.python.org/reference/datamodel#id7 -- Lukasz Rekucki From brett at python.org Mon Jan 11 04:46:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 19:46:04 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > > I don't think ending the 2.x series at 2.7 makes it look bad > > compared to 3.2; it's simply the end of a development line like > > any other software project. I suspect 2.7 will have a protracted > > bugfix window because so much code runs on 2.x exclusively at the > > moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Well yeah; Python 3.0 is just over a year old with 3.1 -- the first robust 3.x release - is about six months old. From the beginning uptake was expected to take years, not months, so saying that 3.x is not popular enough seems premature. > Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > No one has said bugfixes will cease. > > > If there really is an outcry on this we can re-visit the issue, > > but as of right now we need to move forward at some point and 2.7 > > seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. > > I never said that I wanted to make this announcement specifically to provoke outcries. I said if there happened to be outcries we could possibly visit the issue again. Basically I was being considerate and trying to leave the door open to discuss things in the future even though I don't see the situation changing. > IMHO, the release notes should say something like: > > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. > No because that suggests new features will be coming to 2.x which is not going to happen. If you want to say there will be continual bugfix releases for 2.7 as is par the course for Python and that the number of bugfix releases might be more than normal then I am okay with that. > > > trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil > That came and went already a couple months ago when we discussed stopping at 2.6 instead of 2.7 (see the various threads at http://mail.python.org/pipermail/python-dev/2009-November/thread.html). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 05:05:07 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:05:07 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <20100111040507.GA7200@arctrix.com> On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > After the Python 2.7 release, the focus of Python development > > will be on Python 3. There will continue to be maintainance > > releases of Python 2.x. > > No because that suggests new features will be coming to 2.x which is not > going to happen. If you want to say there will be continual bugfix releases > for 2.7 as is par the course for Python and that the number of bugfix > releases might be more than normal then I am okay with that. Are you are saying that if someone steps up to merge the Unladen Swallow features into a 2.8 release and someone also steps up to cut the release that they will be prevented from doing so? Also, are you presuming to channel the BDFL or was this dicussed somewhere other than python-dev? Maybe I'm overreacting but I get the feeling that the larger and less active segment of the Python develpment team hasn't been involved in these decisions. Best regards, Neil From nas at arctrix.com Mon Jan 11 05:17:54 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:17:54 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5C69.7090007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> Message-ID: <20100111041754.GB7200@arctrix.com> On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: > [...] would it still be ok if I closed a 2.x feature request as > "won't fix", or only committed the new feature to the 3.x branch? Yes. Non-bugfix development on 2.x would optional (i.e. done by people who want to spend the time). Since language changes are now out (an initiative I completely support), the majority of non-bugfix changes would be optimizations and platform porting. Regards, Neil From brett at python.org Mon Jan 11 06:06:15 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 21:06:15 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111040507.GA7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 20:05, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > > After the Python 2.7 release, the focus of Python development > > > will be on Python 3. There will continue to be maintainance > > > releases of Python 2.x. > > > > No because that suggests new features will be coming to 2.x which is not > > going to happen. If you want to say there will be continual bugfix > releases > > for 2.7 as is par the course for Python and that the number of bugfix > > releases might be more than normal then I am okay with that. > > Are you are saying that if someone steps up to merge the Unladen > Swallow features into a 2.8 release and someone also steps up to cut > the release that they will be prevented from doing so? > > I honestly don't know, but it's a possibility just like any other new feature request. If people start taking the carrots we have added to 3.x and backporting them to keep the 2.x series alive you are essentially making the 3.x DOA by negating its benefits which I personally don't agree with. > Also, are you presuming to channel the BDFL or was this dicussed > somewhere other than python-dev? This was discussed; see the November 2009 threads. Guido actually suggested ending the 2.x branch at 2.6 until people spoke up about the amount of stuff already backported to 2.7 from 3.x because it was being assumed to be the end of the series to warrant keeping it to help transition to 2.7. > Maybe I'm overreacting but I get > the feeling that the larger and less active segment of the Python > develpment team hasn't been involved in these decisions. > This all came up in November from the 3rd through the 6th (four days) over a ton of email traffic. This was not a snap decision but a heated discussion where even Guido participated that culminated with the decision to stop at 2.7 for the 2.x series; this was not a smoked-filled room decision. I'm sorry if people missed it when they weren't looking, but python-dev is supposed to be the place to watch for this kind of stuff. I don't know how else to bring this stuff to people's attention short of also on python-committers, but developers are basically expected to be on both lists so I don't see where anyone did anything wrong in terms of informing developers. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 06:27:13 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 23:27:13 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: <20100111052713.GA8211@arctrix.com> On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > If people start taking the carrots we have added to 3.x and > backporting them to keep the 2.x series alive you are essentially > making the 3.x DOA by negating its benefits which I personally > don't agree with. I think we have got to the heart of our disagreement. Assume that some superhuman takes all the backwards compatible goodies from 3.x and merges them into 2.x. If that really would kill off Python 3 then I would proclaim it a project that deserved to die. Why should the backwards incompatible changes be endured if they cannot justify their existance by the benefit they provide? I guess I have more confidence in Python 3 than you do. I don't see why Python 2.x needs to be artificially limited so that Python 3 can benefit. Regards, Neil From brett at python.org Mon Jan 11 07:30:49 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 22:30:49 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 21:27, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. > > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. If that really would kill off Python 3 > then I would proclaim it a project that deserved to die. Why should > the backwards incompatible changes be endured if they cannot justify > their existance by the benefit they provide? > > When I said "carrots" I meant everything that makes Python 3 the best version of Python, including the backward-incompatible changes. I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. > I don't view it as artificial but simply a focusing of the development team on what has been determined as the future of Python by Guido and letting go of the past. IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying "Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out". That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Jan 11 08:08:14 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 08:08:14 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <319e029f1001102308p5de87cber2131f3aec1abc9f0@mail.gmail.com> On Mon, Jan 11, 2010 at 06:27, Neil Schemenauer wrote: > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. The superhumans of core developers pretty much already have. That's pretty much 2.7 you are talking about. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From chrism at plope.com Mon Jan 11 08:27:03 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 11 Jan 2010 02:27:03 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4AD2C7.1050703@plope.com> Brett Cannon wrote: > IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev > saying "Python 3 is the future, but we are keeping the old Python 2.x > around because we don't have *that* much faith in the future we have > laid out". That's poison to Python 3 by showing a lack of confidence in > the direction that the BDFL and python-dev as a group has chosen. Now I > could be wrong and there could actually be a large number of active > contributors who want to keep the 2.x series going, but based on the > discussion that occurred the last time this came up I believe the guys > who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort "Python 2". - C From martin at v.loewis.de Mon Jan 11 09:06:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:06:16 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: <4B4ADBF8.3030803@v.loewis.de> Neil Schemenauer wrote: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). I think *that* would give a very bad impression of Python. Depending whom you deal with, the new feature you want may or may not get added to the code base. Contributors would feel even more stranded than they do now, where it may take several years to get a patch reviewed, as you then could submit a patch, and pray that a comitter reviews it who believes in future 2.x releases. The point of setting policies is that it gives every user (contributors, committers, and "end-user" developers) a reliable foundation for expectations. Regards, Martin From arcriley at gmail.com Mon Jan 11 09:06:10 2010 From: arcriley at gmail.com (Arc Riley) Date: Mon, 11 Jan 2010 03:06:10 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AD2C7.1050703@plope.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4AD2C7.1050703@plope.com> Message-ID: after all these years, some people still run AmigaOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 11 09:11:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:11:25 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <4B4ADD2D.6070906@v.loewis.de> > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. Why that? It is a fact: 2.x will not be supported, in some future. Should we lie to users? > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. It shouldn't say that, because it wouldn't be true. Regards, Martin From martin at v.loewis.de Mon Jan 11 09:18:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:18:30 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4ADED6.5080207@v.loewis.de> > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. Because it takes too much time. Too much of my time, but apparently also too much of other people's time. Of course, the less active fraction of Python contributors may not notice, since they just chose to not contribute (which, of course, is fine). However, asking me to work twice as much as I want to on the project to keep two branches alive is just unfair. This has nothing to do with pushing 3.x, but all with managing available manpower and still providing quality software. Regards, Martin From regebro at gmail.com Mon Jan 11 09:42:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 09:42:51 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> This is what it says now: "Python 2.7 is scheduled to be the last major version in the 2.x series before it moves into 5 years of bugfix-only mode. " And during this discussion it has been noted that others than the core python team might pick up Python 2 and make releases. So maybe we can and this discussion by changing that line in future releases to be: "Python 2.7 is scheduled to be the last major version in the 2.x series released by the Python Software Foundation before it moves into 5 years of bugfix-only mode. " That doesn't exclude *others* making a Python 2.8 that could include all sorts of crazy features. :) This is mainly just to get the pointless discussion over with. The current Python core team don't want to make a 2.8, so that will not happen. Someone else might, but as Chris points out, they won't step up until they have to, which is probably at least two years from now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 10:06:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 10:06:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> Message-ID: <4B4AEA25.7010100@v.loewis.de> > "Python 2.7 is scheduled to be the last major version in the 2.x > series released by the Python Software Foundation before it moves into > 5 years of bugfix-only mode. " > > That doesn't exclude *others* making a Python 2.8 that could include > all sorts of crazy features. :) > > This is mainly just to get the pointless discussion over with. I know you are just looking for a compromise, but this shouldn't be it: the PSF has deliberately stayed out of the actual Python engineering, so the release that Benjamin makes is not done by the PSF (but by Benjamin and his contributors :-). I like the wording as it is (although I find the promise of five years of bug fix releases a bit too strong). Regards, Martin From regebro at gmail.com Mon Jan 11 10:19:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 10:19:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AEA25.7010100@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> Message-ID: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" wrote: > I know you are just looking for a compromise, but this shouldn't be it: > the PSF has deliberately stayed out of the actual Python engineering, > so the release that Benjamin makes is not done by the PSF (but by > Benjamin and his contributors :-). Hm. Yeah. That's right of course. I started with saying "official", but then I thought "official according to who?". :) So I changed it to mentioning PSF, but that doesn't work either. I guess the current writing as as good as it gets, unless we change "scheduled" to "expected" or something. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Mon Jan 11 10:42:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:42:05 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: Neil Schemenauer, 11.01.2010 05:17: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). Note that there's also the time it takes to make a release (usually involving at least one beta release), plus the possibility of introducing bugs while adding new features or even while fixing agreed bugs. All of this takes time in addition to the time people might want to invest in 'real' development on the 2.x trunk. Stefan From stephen at xemacs.org Mon Jan 11 10:59:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Jan 2010 18:59:57 +0900 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <8763793qsi.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Schemenauer writes: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. Well, I think it's *worse* than that, and I don't think you really mean "DOA", anyway. (Feel free to correct me, of course.) The problem I see with backporting lots of stuff, and/or adding new features that aren't in 3.0, to 2.x is that it will make 2.x even cruftier, when it was already crufty enough that Guido (and almost all of python-dev) bit the bullet and said "backward compatibility is no excuse for keeping something in 3.0". That surely means that a lot of python-dev denizens will declare non-support 2.x for x > 7. It's not going to be the gradual migration we've seen over the past few months as active people start to spend more and more time on 3 vs. 2; it will be a watershed. Especially if these are new features merged from outside that the "small active segment" doesn't know anything about. From the users' point of view, that amounts to a *fork*, even if it's internal and "friendly". > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. Isn't that a bit ridiculous? I just don't see any evidence that anything like that is going to happen. Worse, if we *assume* it will happen, I don't see any way to assess whether (1) Python 3 goes belly up, (2) there's an effective fork confusing the users and draining the energy of python-dev, or (3) everybody goes "wow!" because it's so cool that everybody wants to keep maintaining an extra 3 branches indefinitely. My opinion is that given the clear direction the "small active segment" is going, telling the users anything but what Brett proposed is disinformation. > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. It's not for Python 3, which you, I, and I'm pretty sure Brett-in-his- heart-of-hearts agree can take care of itself because it is *better* than Python 2. It's for Python, and for the Python community. From walter at livinglogic.de Mon Jan 11 11:37:56 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 11:37:56 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001091438.43576.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> Message-ID: <4B4AFF84.7070206@livinglogic.de> On 09.01.10 14:38, Victor Stinner wrote: > Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : >>> Good idea, I choosed open(filename, encoding="BOM"). >> >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Why not, this is another solution to the point (2) (Check for a BOM while > reading or detect it before?). Which encoding would be used if there is not > BOM? UTF-8 sounds like a good choice. UTF-8 might be a good choice, are the failback could be specified in the encoding name, i.e. open("file.txt", encoding="BOM-UTF-8") falls back to UTF-8, if there's no BOM at the start. This could be implemented via a custom codec search function (see http://docs.python.org/library/codecs.html#codecs.register for more info). Servus, Walter From regebro at gmail.com Mon Jan 11 12:12:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:12:20 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4AFF84.7070206@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> Message-ID: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: > UTF-8 might be a good choice No, fallback if there is no BOM should be the local settings, just as fallback is today if you don't specify a codec. I mean, what if you want to look for a BOM but fall back to something else? How far will we go with encoding special information in the codecs names? codec='BOM else UTF-16 else locale'? :-) BOM is not a locale, and should not be a locale. Having a locale called BOM is wrong per se. It should either be default to look for a BOM when codec=None, or a special parameter. If none of these are desired, then we need a special function that takes a filename or file handle, and looks for a BOM and returns the codec found or None. But I find that much less natural and obvious than checking the BOM when codec=None. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Mon Jan 11 12:13:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:13:00 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <319e029f1001110313u1d839deodfe06a6c7ec423cf@mail.gmail.com> On Mon, Jan 11, 2010 at 12:12, Lennart Regebro wrote: > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. D'oh! I mean codec here obviously. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 13:29:04 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 13:29:04 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4913DF.7050801@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> Message-ID: <4B4B1990.90705@livinglogic.de> On 10.01.10 00:40, "Martin v. L?wis" wrote: >>> How does the requirement that it be implemented as a codec miss the >>> point? >> >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. > > Yes - however, Victor currently apparently *doesn't* want it to be the > default, but wants the user to specify encoding="BOM". If so, it isn't > the default, and it is easy to implement as a codec. > >>> FWIW, I agree with Walter that if it is provided through the encoding= >>> argument, it should be a codec. If it is built into the open function >>> (for whatever reason), it must be provided by some other parameter. >> >> Why not simply encoding=None? > > I don't mind. Please re-read Walter's message - it only said that > *if* this is activated through encoding="BOM", *then* it must be > a codec, and could be on PyPI. I don't think Walter was talking about > the case "it is not activated through encoding='BOM'" *at all*. However if this autodetection feature is useful in other cases (no matter how it's activated), it should be a codec, because as part of the open() function it isn't reusable. >> The default value should provide the most useful >> behaviour possible. Forcing users to choose between two different autodetection >> strategies (encoding=None and another one) is a little insane IMO. And encoding="mbcs" is a third option on Windows. > That wouldn't disturb me much. There are a lot of things in that area > that are a little insane, starting with Microsoft Windows :-) ;) Servus, Walter From barry at python.org Mon Jan 11 13:37:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 07:37:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5DCE.3070909@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: On Jan 10, 2010, at 6:07 PM, Martin v. L?wis wrote: > As for decisions: I don't think there was an official BDFL pronouncent, > but I recall Guido posting a message close to that, proposing that 2.7 > will be a release that will see bug fix releases for an indefinite > period of time (where indefinite != infinite). This was shortly after > him proposing that perhaps we shouldn't make a 2.7 release at all, and > stop at 2.6. IMO, the focus for Python 2.7 (and beyond) must be on helping people transition to Python 3. That should be the reason why a 2.7 is released and, what should dictate whether a 2.8 is necessary. Maybe everything people need (except manpower and round tuits) is already there. I was pleasantly surprised to find that Distribute supports automatically running 2to3 at install time for Python packages. This allowed me to "port" a recent (admittedly small) package to Python 3 by making it "python2.6 -3" clean and adding a couple of attributes to my setup.py[1]. I don't know how widely that feature's known, but I think it's *huge*, and exactly what we need to be doing. The more we can make it easy for people to port their Python 2 code to Python 3, and to support that with one code base, the quicker we'll get to a predominantly Python 3 world. So the question we should be asking is: what's missing from Python 2.7 to help with this transition? If we can't get it into 2.7, then why, and would pushing it back to 2.8 help at all? -Barry [1] modulo a bug in Distribute that caused doctest in separate files to not be used when running under Python 3. From solipsis at pitrou.net Mon Jan 11 13:39:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 13:39:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <1263213574.3327.0.camel@localhost> > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. It is reusable as part of io.TextIOWrapper, though. Regards Antoine. From regebro at gmail.com Mon Jan 11 13:45:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 13:45:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. But an autodetect feature is not a codec. Sure it should be reusable, but making it a codec seems to be a weird hack to me. And how would you reuse it if it was a codec? A reusable autodetect feature would be useable to detect what codec it is. A autodetect codec would not be useful for that, as it would simply just decode. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 14:21:15 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 14:21:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B25CB.5030003@livinglogic.de> On 11.01.10 13:45, Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: >> However if this autodetection feature is useful in other cases (no >> matter how it's activated), it should be a codec, because as part of the >> open() function it isn't reusable. > > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. I think we already had this discussion two years ago in the context of XML decoding ;): http://mail.python.org/pipermail/python-dev/2007-November/075138.html > And how would > you reuse it if it was a codec? A reusable autodetect feature would be > useable to detect what codec it is. A autodetect codec would not be > useful for that, as it would simply just decode. I have implemented an XML codec (as part of XIST: http://pypi.python.org/pypi/ll-xist), that can do that: >>> from ll import xml_codec >>> import codecs >>> c = codecs.getincrementaldecoder("xml")() >>> c.encoding >>> c.decode(">> c.encoding >>> c.decode(" version='1.0'") u'' >>> c.encoding >>> c.decode(" encoding='iso-8859-1'?>") u"" >>> c.encoding 'iso-8859-1' Servus, Walter From regebro at gmail.com Mon Jan 11 14:47:12 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 14:47:12 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B25CB.5030003@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B25CB.5030003@livinglogic.de> Message-ID: <319e029f1001110547n1d1c9831p34b0eac519d13f9d@mail.gmail.com> On Mon, Jan 11, 2010 at 14:21, Walter D?rwald wrote: > I think we already had this discussion two years ago in the context of > XML decoding ;): Yup. Ans Martins answer then is my answer now: "> So the code is good, if it is inside an XML parser, and it's bad if it > is inside a codec? Exactly so. This functionality just *isn't* a codec - there is no encoding. Instead, it is an algorithm for *detecting* an encoding." The conclusion was that a method do autodetect encodings would be good. I think the same conclusion applies here. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 18:16:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:16:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B5CF5.50806@v.loewis.de> > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. Well, the existing UTF-16 codec also is an autodetect feature (to detect the endianness), and I don't consider it a weird hack. Regards, Martin From regebro at gmail.com Mon Jan 11 18:27:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:27:01 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B5CF5.50806@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> Message-ID: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> On Mon, Jan 11, 2010 at 18:16, "Martin v. L?wis" wrote: >> But an autodetect feature is not a codec. Sure it should be reusable, >> but making it a codec seems to be ?a weird hack to me. > > Well, the existing UTF-16 codec also is an autodetect feature (to > detect the endianness), and I don't consider it a weird hack. So the BOM codec should raise a UnicodeDecodeError if there is no BOM? Because that's what it would have to do, in that case, because it can't fall back on anything, it has to handle and implement all encodings that have a BOM. And is it then actually very useful? You would have to do a try/except first with encoding='BOM' and then encoding=None to get the fallback to the standard. I must say that I find this whole thing pretty obvious. 'BOM' is not an encoding. Either there should be a method to get the encoding from the BOM, returning None of there isn't one, or open() should look at the BOM when you pass in encoding=None. Or both. That covers all usecases, is easy and obvious. Either open(file=foo, encoding=None) or open(file, encoding=encoding_from_bom(file)) I can't see that open(file, encoding='BOM') has any benefit over this, covers any extra usecase and is clearer in any way. Instead it adds something confusing: An encoding that isn't an encoding. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From python at mrabarnett.plus.com Mon Jan 11 18:35:35 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:35:35 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <4B4B6167.40606@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: >> UTF-8 might be a good choice > > No, fallback if there is no BOM should be the local settings, just as > fallback is today if you don't specify a codec. > I mean, what if you want to look for a BOM but fall back to something > else? How far will we go with encoding special information in the > codecs names? codec='BOM else UTF-16 else locale'? :-) > > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. It should either be default to look for a > BOM when codec=None, or a special parameter. If none of these are > desired, then we need a special function that takes a filename or file > handle, and looks for a BOM and returns the codec found or None. But > I find that much less natural and obvious than checking the BOM when codec=None. > Or pass a function that accepts a byte stream or the first few bytes and returns the encoding and any unused bytes (because the byte stream might not be seekable)? def guess_encoding(byte_stream): data = byte_stream.read(2) if data == b"\xFE\xFF": return "UTF-16BE", b"" return "UTF-8", data text_file = open(filename, encoding=guess_encoding) ... From python at mrabarnett.plus.com Mon Jan 11 18:46:30 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:46:30 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> Message-ID: <4B4B63F6.1020309@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" > wrote: >> I know you are just looking for a compromise, but this shouldn't be >> it: the PSF has deliberately stayed out of the actual Python >> engineering, so the release that Benjamin makes is not done by the >> PSF (but by Benjamin and his contributors :-). > > Hm. Yeah. That's right of course. I started with saying "official", > but then I thought "official according to who?". :) "Official" is whatever the BDFL says it is! :-) > So I changed it to mentioning PSF, but that doesn't work either. I > guess the current writing as as good as it gets, unless we change > "scheduled" to "expected" or something. > From martin at v.loewis.de Mon Jan 11 18:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:59:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> Message-ID: <4B4B66EC.7000203@v.loewis.de> > I must say that I find this whole thing pretty obvious. 'BOM' is not > an encoding. That I certainly agree with. > That covers all usecases, is easy and obvious. Either open(file=foo, > encoding=None) or open(file, encoding=encoding_from_bom(file)) > > I can't see that open(file, encoding='BOM') has any benefit over this, Well, it would have the advantage that Walter pointed out: you can implement it independent of the open() implementation, and even provide it in older versions of Python. Regards, Martin From regebro at gmail.com Mon Jan 11 18:59:52 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B63F6.1020309@mrabarnett.plus.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> Message-ID: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> On Mon, Jan 11, 2010 at 18:46, MRAB wrote: > "Official" is whatever the BDFL says it is! :-) Heh, right. So, it should say "Guido wants 2.7 to be the last main version of Python 2, so it probably will be. We promise to release bugfixes it for, like, ages". No need to be needlessly formal. :-D -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From olemis at gmail.com Mon Jan 11 19:58:01 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 13:58:01 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> [...] > I had similar issues too (please read below ;o) ... On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > About guessing the encoding, I experienced this issue while I was developing a Trac plugin. What I was doing is as follows : - I guessed the MIME type + charset encoding using Trac MIME API (it was a CSV file encoded using UTF-16) - I read the file using `open` - Then wrapped the file using `codecs.EncodedFile` - Then used `csv.reader` ... and still get the BOM in the first value of the first row in the CSV file. {{{ #!python >>> mimetype 'utf-16-le' >>> ef = EncodedFile(f, 'utf-8', mimetype) }}} IMO I think I am +1 for leaving `open` just like it is, and use module `codecs` to deal with encodings, but I am strongly -1 for returning the BOM while using `EncodedFile` (mainly because encoding is explicitly supplied in ;o) > --Guido > CMIIW anyway ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mal at egenix.com Mon Jan 11 21:44:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Jan 2010 21:44:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> Message-ID: <4B4B8DB2.1060102@egenix.com> Olemis Lang wrote: >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>> BOM should be "ignored". >>> > [...] >> > > I had similar issues too (please read below ;o) ... > > On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> > > About guessing the encoding, I experienced this issue while I was > developing a Trac plugin. What I was doing is as follows : > > - I guessed the MIME type + charset encoding using Trac MIME API (it > was a CSV file encoded using UTF-16) > - I read the file using `open` > - Then wrapped the file using `codecs.EncodedFile` > - Then used `csv.reader` > > ... and still get the BOM in the first value of the first row in the CSV file. You didn't say, but I presume that the charset guessing logic returned either 'utf-16-le' or 'utf-16-be' - those encodings don't remove the leading BOM. The 'utf-16' codec will remove the BOM. > {{{ > #!python > >>>> mimetype > 'utf-16-le' >>>> ef = EncodedFile(f, 'utf-8', mimetype) > }}} Same here: the UTF-8 codec will not remove the BOM, you have to use the 'utf-8-sig' codec for that. > IMO I think I am +1 for leaving `open` just like it is, and use module > `codecs` to deal with encodings, but I am strongly -1 for returning > the BOM while using `EncodedFile` (mainly because encoding is > explicitly supplied in ;o) Note that EncodedFile() doesn't do any fancy BOM detection or filtering. This is the job of the codecs. Also note that BOM removal is only valid at the beginning of a file. All subsequent BOM-bytes have to be read as-is (they map to a zero-width non-breaking space) - without removing them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 11 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Jan 11 22:12:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 07:12:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <4B4B9447.4060101@gmail.com> Benjamin Peterson wrote: > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors I would call it a documentation bug: by leaving out the __get__ you're telling Python to override *just* the setting of the attribute, while still using the normal lookup process for retrieving the attribute (i.e. allowing the attribute to be shadowed in the instance dictionary). Adding a "print('Descriptor Invoked')" to the __set__ method in your example: >>> x = X() >>> print(x.attr) <__main__.Descr object at 0x7f283b51ac50> >>> x.attr = 42 Descriptor invoked >>> print(x.attr) 42 >>> x.attr = 6*9 Descriptor invoked >>> print(x.attr) 54 Note that the behaviour here is still different from that of a data descriptor: with a data descriptor, once it gets shadowed in the instance dictionary, the descriptor is ignored *completely*. The only way to get the descriptor involved again is to eliminate the shadowing. The non-data descriptor with only __set__ is just choosing not to override the attribute lookup process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Mon Jan 11 22:29:38 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 16:29:38 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B8DB2.1060102@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> <4B4B8DB2.1060102@egenix.com> Message-ID: <24ea26601001111329i7f609aa1i9f5db7eb61f1fae7@mail.gmail.com> Probably one part of this is OT , but I think it could complement the discussion ;o) On Mon, Jan 11, 2010 at 3:44 PM, M.-A. Lemburg wrote: > Olemis Lang wrote: >>> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >>> wrote: >>>> Hi, >>>> >>>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>>> BOM should be "ignored". >>>> >> [...] >>> >> >> I had similar issues too (please read below ;o) ... >> >> On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >>> >> >> About guessing the encoding, I experienced this issue while I was >> developing a Trac plugin. What I was doing is as follows : >> >> - I guessed the MIME type + charset encoding using Trac MIME API (it >> was a CSV file encoded using UTF-16) >> - I read the file using `open` >> - Then wrapped the file using `codecs.EncodedFile` >> - Then used `csv.reader` >> >> ... and still get the BOM in the first value of the first row in the CSV file. > > You didn't say, but I presume that the charset guessing logic > returned either 'utf-16-le' or 'utf-16-be' Yes. In fact they return the full mimetype 'text/csv; charset=utf-16-le' ... ;o) > - those encodings don't > remove the leading BOM. ... and they should ? > The 'utf-16' codec will remove the BOM. > In this particular case there's nothing I can do, I have to process whatever charset is detected in the input ;o) >> {{{ >> #!python >> >>>>> mimetype >> 'utf-16-le' >>>>> ef = EncodedFile(f, 'utf-8', mimetype) >> }}} > > Same here: the UTF-8 codec will not remove the BOM, you have > to use the 'utf-8-sig' codec for that. > >> IMO I think I am +1 for leaving `open` just like it is, and use module >> `codecs` to deal with encodings, but I am strongly -1 for returning >> the BOM while using `EncodedFile` (mainly because encoding is >> explicitly supplied in ;o) > > Note that EncodedFile() doesn't do any fancy BOM detection or > filtering. ... directly. > This is the job of the codecs. > OK ... to come back to the scope of the subject, in the general case, I think that BOM (if any) should be handled by codecs, and therefore indirectly by EncodedFile . If that's a explicit way of working with encodings I'd prefer to use that wrapper explicitly in order to (encode | decode) the file and let the codec detect whether there's a BOM or not and ?adjust? `tell`, `read` and everything else in that wrapper (instead of `open`). > Also note that BOM removal is only valid at the beginning of > a file. All subsequent BOM-bytes have to be read as-is (they > map to a zero-width non-breaking space) - without removing them. > ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Test cases for custom query (i.e report 9) ... PASS (1.0.0) - http://simelo.hg.sourceforge.net/hgweb/simelo/trac-gviz/rev/d276011e7297 From martin at v.loewis.de Mon Jan 11 22:42:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:42:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: <4B4B9B55.1010709@v.loewis.de> > So the question we should be asking is: what's missing from Python > 2.7 to help with this transition? Wrt. the distribute feature you've noticed: Python 3.0 already supports that in distutils. So from day one of Python 3.x, you could have used that approach for porting Python code. I had been promoting it ever since, but it's taking up only slowly, partially because it has competition in the approaches "burn the bridges" (i.e. convert to 3.x once, and then have two code bases, or abandon 2.x), and "avoid 2to3" (i.e. try to write code that works in both 2.x and 3.x unmodified). > If we can't get it into 2.7, then > why, and would pushing it back to 2.8 help at all? I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: a) telling people that they have to move to 2.6 first actually hurts migration, instead of helping, because it implies to them that they have to drop old versions (e.g. 2.3.) - just because they had *always* dropped old versions before supporting new ones. b) IMO, people also don't gain much by first migrating to 2.6. In principle, it gives them the opportunity to get py3k warnings. However, I haven't heard a single positive report where these warnings have actually helped people in porting. Yours is the first report saying that you followed the official guideline, but you didn't state whether doing so actually helped (or whether you just ported to 2.6 because the guideline told you to). c) whatever 2.7 does (except perhaps for the warnings), it won't be useful to applications, because they couldn't use it, anyway: they need to support 2.4 and 2.5, and it won't have any of the gimmicks people come up with for 2.7. Adding gimmicks to 2.7 might actually hurt porting: people may have to special-case 2.7 because their work-arounds for older versions may break in 2.7 (e.g. testing whether a name is *not* defined, when it becomes defined in 2.7), plus it gives them an incentive to not port yet until they can drop support for anything before 2.7. Inherently, 2.8 can't improve on that. Regards, Martin From martin at v.loewis.de Mon Jan 11 22:44:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:44:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> Message-ID: <4B4B9BB8.3070505@v.loewis.de> > So, it should say "Guido wants 2.7 to be the last main version of > Python 2, so it probably will be. We promise to release bugfixes it > for, like, ages". > > No need to be needlessly formal. :-D That summarizes my understanding of what Guido said fairly well :-) +1 for putting it into the announcements. Regards, Martin From fuzzyman at voidspace.org.uk Mon Jan 11 23:11:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 11 Jan 2010 22:11:44 +0000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4B9447.4060101@gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> Message-ID: <4B4BA220.20701@voidspace.org.uk> On 11/01/2010 21:12, Nick Coghlan wrote: > Benjamin Peterson wrote: > > My question is: Is this a doc bug or a implementation bug? If the > >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors >> > [snip...] > > Note that the behaviour here is still different from that of a data > descriptor: with a data descriptor, once it gets shadowed in the > instance dictionary, the descriptor is ignored *completely*. The only > way to get the descriptor involved again is to eliminate the shadowing. > The non-data descriptor with only __set__ is just choosing not to > override the attribute lookup process. > > Does that mean we need a third class of descriptors that are neither data descriptors nor non-data descriptors? What should we call them: really-not-data-descriptors? All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Mon Jan 11 23:55:01 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Jan 2010 14:55:01 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9BB8.3070505@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: +1 from me. (With the caveat that "time will tell" is definitely the ultimate answer. Plans may change unexpectedly, even though we don't expect them to.) PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not helpful. But I trust he has ported a lot more code to 3.x than I have personally. Are there other experiences? --Guido On Mon, Jan 11, 2010 at 1:44 PM, "Martin v. L?wis" wrote: >> So, it should say "Guido wants 2.7 to be the last main version of >> Python 2, so it probably will be. We promise to release bugfixes it >> for, like, ages". >> >> No need to be needlessly formal. :-D > > That summarizes my understanding of what Guido said fairly well :-) > > +1 for putting it into the announcements. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Jan 12 00:16:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 00:16:47 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <4B4BB15F.5020807@v.loewis.de> Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. That's really because I haven't even attempted to use it. Instead, the software would fall flat on its own when running the test suite in 3.x. IOW, I don't need 2.6 to tell me what might break when I can see 3.x actually breaking. Regards, Martin From david.lyon at preisshare.net Tue Jan 12 00:22:42 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:22:42 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Hi Martin, > Of course, the less active fraction of Python contributors may not > notice, since they just chose to not contribute (which, of course, > is fine). However, asking me to work twice as much as I want to > on the project to keep two branches alive is just unfair. Totally true. Actually as an end-developer I'd say that python 2.x series from a programming perspective is quite good. It doesn't need the addition of a lot of new features imho. So for me, I think too much time spent there would be not yield great benefits. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Well, we all know your work is super quality. :-) That's not being contested. However, Quality can be measured different ways and it can be assessed in different ways. Quality itself is a subjective thing. The point I'm only making is that if a piece of software doesn't have "new" things added over time, then users can get a reverse impression of a lack of quality. We've all seen where 'internal' quality can increase and user perceptions can decrease. It could be things like improved graphics and things readily apparent to the user. At the moment, I would say that the "internal" quality of the python 2.x series is super high. "external" quality issues such as the packaging dilemma give the user the opposite quality experience. But people are working on that as best they can elsewhere. I'll leave it at that. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Python 3.x needs more carrots. >From an ordinary (perphaps ignorant) user perspective there is nothing. Yes, we know if we actually will start programming then we will like it more. But my wishes to Santa Claus would be allow the free flow of PEPs for Python 3 packaging. Even encourage it. As an end developer, here's what I'd like from Santa in 2010 to get me to swap to python 3: * get all the packages on pypi tested for python 3 * put a web based package manager in python 3. This would perhaps be based around PIP (keep many people happy) and would look much like the built in web-console that you get with a $200 router. * Incorporate SCM based end-user software installs by adding to python3-stdlib the SCM support packages (cvs,bzr,svn,hg). This would *really* help in the scientific community. * put a web interface on distutils also so that we don't have to use a command line interface. (I want a pic of a smiley girl to great me when I build something - "Are you ready to build now Honey?"). ok - I joke. But the point is made. So, ok, maybe these things aren't about 'code' quality. But rather user experience. Things like these do count also as "quality" via the technical term "perception of quality". If the PEP process is as unblocked as the documentation implies, implying that anybody can contribute to Python 3. Then there shouldn't be any issue. David From solipsis at pitrou.net Tue Jan 12 00:37:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 23:37:47 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon preisshare.net> writes: > > > This has nothing to do with pushing 3.x, but all with managing > > available manpower and still providing quality software. > > Python 3.x needs more carrots. As someone who experiences the difference almost every day, I can say 3.x definitely has enough carrots for me. The simple fact that I will be able to raise exceptions with non-ASCII error messages without having to workaround a hopelessly broken conversion/reporting logic is a huge improvement. And that's only a small part of the picture. Regards Antoine. From janssen at parc.com Tue Jan 12 00:47:55 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 11 Jan 2010 15:47:55 PST Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <17215.1263253675@parc.com> > David Lyon preisshare.net> writes: > > > > > This has nothing to do with pushing 3.x, but all with managing > > > available manpower and still providing quality software. > > > > Python 3.x needs more carrots. I'd be happy to move UpLib to Python 3, when the various packages that I need are also on Python 3. And that depresses me to think about. I need Medusa ReportLab PyLucene Email Vobject Mutagen Pyglet Hachoir Win32 I'm on the mailing lists for a lot of these, and the only one that I know is thinking about Python 3 is Email. I'd expect Win32 and PIL to also be thinking about it, but I haven't heard anything. Bill From david.lyon at preisshare.net Tue Jan 12 00:47:51 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:47:51 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <1220.218.214.45.58.1263253671.squirrel@syd-srv02.ezyreg.com> Antoine writes: > As someone who experiences the difference almost every day, I can say 3.x > definitely has enough carrots for me. The simple fact that I will be able > to > raise exceptions with non-ASCII error messages without having to > workaround a > hopelessly broken conversion/reporting logic is a huge improvement. And > that's > only a small part of the picture. In no way could I disagree with you. Ascii works fine for us here - but I guess we're lucky. Just keep pushing python 3 and have a nice day.. David From barry at python.org Tue Jan 12 01:11:28 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:11:28 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100111191128.39020d89@freewill> On Jan 11, 2010, at 10:42 PM, Martin v. L?wis wrote: >Wrt. the distribute feature you've noticed: Python 3.0 already supports >that in distutils. So from day one of Python 3.x, you could have used >that approach for porting Python code. I had been promoting it ever >since, but it's taking up only slowly, partially because it has >competition in the approaches "burn the bridges" (i.e. convert to 3.x >once, and then have two code bases, or abandon 2.x), and "avoid 2to3" >(i.e. try to write code that works in both 2.x and 3.x unmodified). Interesting. The only reason I never used it before is because I didn't know about it. Am I alone? Maybe I'm projecting my own preferences, but it seems to me that supporting both Python 2 and 3 from the same code base would be a very powerful way to enable a large amount of existing code to support Python 3. I'm certainly going to do this from now on with all the libraries I maintain. I don't have any cycles to write code twice and I can't abandon Python 2 yet. I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. As an example, the one library I've already ported used a metaclass. I don't see any way to specify that the metaclass should be used in a portable way. In Python 2.6 it's: class Foo: __metaclass__ = Meta and in Python 3 it's: class Foo(metaclass=Meta): 2to3 made that pain go away. >I've done a fair bit of 3.x porting, and I'm firmly convinced that >2.x can do nothing: > >a) telling people that they have to move to 2.6 first actually > hurts migration, instead of helping, because it implies to them > that they have to drop old versions (e.g. 2.3.) - just because > they had *always* dropped old versions before supporting new ones. Is it just an implication, or is it reality? I have yet to try to support older versions of Python and also support Python 3. (The one package I've done this with so far only supports Python 2.6.) >b) IMO, people also don't gain much by first migrating to 2.6. > In principle, it gives them the opportunity to get py3k warnings. > However, I haven't heard a single positive report where these > warnings have actually helped people in porting. Yours is the > first report saying that you followed the official guideline, > but you didn't state whether doing so actually helped (or whether > you just ported to 2.6 because the guideline told you to). Python 2.6 has other useful features, which I want to take advantage of, so getting py3k warnings is a bonus. Running 'python2.6 -3 setup.py test' over my code did in fact help clean up a couple of problems. Seems like a good first step when considering Python 3 support to me. >c) whatever 2.7 does (except perhaps for the warnings), it won't > be useful to applications, because they couldn't use it, anyway: > they need to support 2.4 and 2.5, and it won't have any of the > gimmicks people come up with for 2.7. Adding gimmicks to 2.7 > might actually hurt porting: people may have to special-case > 2.7 because their work-arounds for older versions may break in 2.7 > (e.g. testing whether a name is *not* defined, when it becomes > defined in 2.7), plus it gives them an incentive to not port > yet until they can drop support for anything before 2.7. > >Inherently, 2.8 can't improve on that. I'm not so sure. Yes, as a package maintainer there are older versions to think about, but time moves on for everyone (hopefully :). By the time 2.8 is released, what will be the minimum version of Python provided by most OS vendors (where the majority of Python users probably get their 'python')? I guess some people will have to support everything from Python 2.3 to 2.8 but you're talking supporting something like a spread of 7 years of Python versions. What other platform do you support for 7 years? Even Ubuntu long term support (LTS) releases only live for 5 years and even then, newer Pythons are often back ported. Or if not, then who cares? You're not going to use a newer version of a library on an LTS anyway. I'm not necessarily saying that justifies a 2.8 release. I'm just asking how we can make it easier for people to port to Python 3. The automatic 2to3 has already made it easier for me to port one library to Python 3 because I barely had to even think about it. Maybe that's enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 01:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:12:19 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <20100111191219.5fdd2542@freewill> On Jan 11, 2010, at 02:55 PM, Guido van Rossum wrote: >PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not >helpful. But I trust he has ported a lot more code to 3.x than I have >personally. Are there other experiences? I've only done one small library so far, but it was helpful to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From andrew at bemusement.org Tue Jan 12 01:07:26 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Tue, 12 Jan 2010 11:07:26 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100112000726.GO32351@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > I've done a fair bit of 3.x porting, and I'm firmly convinced that > 2.x can do nothing: [...] > Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. From solipsis at pitrou.net Tue Jan 12 01:28:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 00:28:26 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: Andrew Bennetts bemusement.org> writes: > > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. I don't think a 2.8 could make you much closer to 3.x. AFAICT, every possible way to ease transition to 3.x will already be in 2.7, or almost. But perhaps I'm lacking imagination. Regards Antoine. From brian.curtin at gmail.com Tue Jan 12 02:57:46 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 11 Jan 2010 19:57:46 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > -Brett > I think it would be interesting to see how people are using the tracker, or how they want to be using it. For example, there are currently over 1500 open issues with no stage set, some of which seemingly haven't been read by anyone at all. Would a properly set stage field save issues from falling into a black hole? Food for thought: according to the last tracker summary, there are over 1000 open issues with a patch, and issues stay open an average of 700 days (median 450). Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Jan 12 04:53:24 2010 From: jackdied at gmail.com (Jack Diederich) Date: Mon, 11 Jan 2010 22:53:24 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. [sidebar] 1) the metaclass fixer was a PITA to implement. 2) 95% of __metaclass__ definitions searchable via google code were of the "__metaclass__ = type" variety. The 2to3 patch exists only because of the few other uses. 3) 100% of the module level assignments in public projects were the "__metaclass__ = type" variety which is why there isn't a fixer for that. Also, a fixer would have been really, really ugly (munge every class definition in this module because there is a top level assignment). -Jack From benjamin at python.org Tue Jan 12 05:01:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 11 Jan 2010 22:01:40 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <1afaf6161001112001t5e7bab83t7d93f33fa11ce849@mail.gmail.com> 2010/1/11 Jack Diederich : > On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: >> As an example, the one library I've already ported used a metaclass. ?I don't >> see any way to specify that the metaclass should be used in a portable way. >> In Python 2.6 it's: >> >> class Foo: >> ? ?__metaclass__ = Meta >> >> and in Python 3 it's: >> >> class Foo(metaclass=Meta): >> >> 2to3 made that pain go away. > > [sidebar] > 1) the metaclass fixer was a PITA to implement. Does this make it any less useful, though? :) -- Regards, Benjamin From steven.bethard at gmail.com Tue Jan 12 06:57:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 11 Jan 2010 21:57:18 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 4:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. Actually there's a solution to this one too: FooBase = Meta('FooBase', (), {}) class Foo(FooBase): ... That should work in Python 2.X and 3.X. I've got argparse running on Python 2.3-3.1, and the changes were pretty easy. You can see them all in the revision here: http://code.google.com/p/argparse/source/detail?r=12 I have aspirations of putting all of the tricks I learned up up on the Wiki somewhere, but I just haven't had the time. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From regebro at gmail.com Tue Jan 12 07:30:10 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:30:10 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> On Mon, Jan 11, 2010 at 23:55, Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. But I trust he has ported a lot more code to 3.x than I have > personally. Are there other experiences? It doesn't warn for that many of the unportable problems, but I'm not sure it can warn for them either. It's certainly helpful, just not very much. :) I think the biggest help was added in 2.6.2, and that's warning for old integer division. It will also warn for modules that aren't supported anymore, if I remember correctly, and that's often helpful. But it won't warn for real tricky problems, like binary vs text in strings, and I don't see how it can either. And, I just realized, it doesn't warn for you using cmp or __cmp__ either, and 2to3 won't fix that, so it should actually warn for it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:49:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:49:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> On Tue, Jan 12, 2010 at 01:11, Barry Warsaw wrote: > Interesting. ?The only reason I never used it before is because I didn't know > about it. ?Am I alone? Maybe not, but the Distribute feature is there because IMO the distutils feature by itself isn't particularily useful. You need to write your own distutils extensions, in practice, and they are not trivial. Distribute has simply done it for you. :) > I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:53:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:53:20 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <319e029f1001112253x511f8109ja2300a022010ef99@mail.gmail.com> On Tue, Jan 12, 2010 at 01:07, Andrew Bennetts wrote: > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. ?I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. When 2.7 was discussed several people agreed that 2.7 should include as much 3.x stuff as possible to ease transition. That turned out to not be very much, so I'm not sure there is more. :) Unless of course, 2.8 starts including more of the refactored libraries, but that's a very minor issue in porting, so it won't help much. To really help, it needs to start implement things that break backwards compatibility. That would be weird. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Tue Jan 12 10:39:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:39:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4BA220.20701@voidspace.org.uk> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> <4B4BA220.20701@voidspace.org.uk> Message-ID: <4B4C435B.7080903@gmail.com> Michael Foord wrote: >> Note that the behaviour here is still different from that of a data >> descriptor: with a data descriptor, once it gets shadowed in the >> instance dictionary, the descriptor is ignored *completely*. The only >> way to get the descriptor involved again is to eliminate the shadowing. >> The non-data descriptor with only __set__ is just choosing not to >> override the attribute lookup process. >> >> > > Does that mean we need a third class of descriptors that are neither > data descriptors nor non-data descriptors? Not really - leaving out __get__ when defining __set__ just creates a data descriptor that happens to use the default attribute look-up process rather than defining a different one. (Note that I had the data/non-data terminology backwards in my previous message - I tend to think of the two kinds of descriptor as enforced and non-enforced respectively precisely because I have to think about it to remember that "functions are non-data descriptors and properties are data descriptors, hence non-data descriptors only define __get__ while data descriptors define __set__". I don't find the data/non-data terminology intuitive at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:44:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:44:18 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> Message-ID: <4B4C4472.10104@gmail.com> Lennart Regebro wrote: > And, I just realized, it doesn't warn for you using cmp or __cmp__ > either, and 2to3 won't fix that, so it should actually warn for it. I have a vague recollection that we tried to warn for that and ended up nixing the warning due to vast swarms of false alarms (or because you couldn't write correct 2.x code without triggering it). I'd be happy for someone to prove my recollection wrong though (i.e. by writing a patch that implements such warnings). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:48:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:48:57 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4C4589.70906@gmail.com> David Lyon wrote: >> This has nothing to do with pushing 3.x, but all with managing >> available manpower and still providing quality software. > > Python 3.x needs more carrots. As Guido has said a few times, the gains are far greater for *new* Python developers than they are for existing ones. Existing developers have to unlearn old habits and wait for libraries they already use to get ported. New developers can just get started with a much cleaner language. They don't have as rich a 3rd party ecosystem on Python 3 as they would on Python 2.x at this point in time, but unlike existing developers they also have the luxury of cherry-picking just those packages that already have Python 3 support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Jan 12 11:20:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:20:27 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit?: > > For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. I think most issues /have/ been read. It's just that for many of them, nobody is interested enough in or feels competent enough for fixing them. > Would a properly set stage field save issues from > falling into a black hole? I don't think this has anything to do with properly setting the stage field. We just have limited time and manpower. Perhaps one of our goals should be to reach out more to potential contributors. > Food for thought: according to the last tracker summary, there are over > 1000 open issues with a patch, and issues stay open an average of 700 > days (median 450). As for open issues with a patch, a "code review" button sending this patch to Rietveld (or any other similar tool) is something many of us have been hoping for :-) It would make reviewing easier, faster and more thorough (because you visualize things better than by looking at a diff). Regards Antoine. From solipsis at pitrou.net Tue Jan 12 11:36:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:36:49 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Tue, 12 Jan 2010 10:20:27 +0000, Antoine Pitrou a ?crit?: > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Speaking of which, Steve had something to say about it: http://holdenweb.blogspot.com/2009/11/comments-or-not-public-or- private.html cheers Antoine. From ncoghlan at gmail.com Tue Jan 12 13:10:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 22:10:14 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C66A6.2040601@gmail.com> Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : >> For example, there are currently over >> 1500 open issues with no stage set, some of which seemingly haven't been >> read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. There are actually a whole host of reasons issues can stagnate: - a feature request may seem reasonable (hence it doesn't get rejected outright), but the right API may not be clear (hence it doesn't get implemented in the near term) - a patch may be reviewed and found to have significant defects (or simply be overreaching the stated goal) but the original patch poster loses interest after meeting resistance in their ambition to fix something that is "obviously" broken - a problem may simply be hard to fix in a backwards compatible way (or even at all!) Ultimately it boils down to Antoine's two categories (lack of interest and lack of relevant expertise to make a final decision) but there are a lot of different ways to get into those two buckets. Because we aren't ruthless about pruning those kinds of issues out of the tracker they're the ones that are going to accumulate over time. I'd actually be interested to know what the issue stats are like when RFEs are excluded and when the search is limited to the items flagged as 'easy'. If easy bug fix issues are taking a long time to get closed than that would bother me a lot more than RFEs that sit on the tracker for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 12 13:14:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:14:47 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071447.675c8f24@freewill> On Jan 11, 2010, at 10:53 PM, Jack Diederich wrote: >3) 100% of the module level assignments in public projects were the >"__metaclass__ = type" variety which is why there isn't a fixer for >that. Also, a fixer would have been really, really ugly (munge every >class definition in this module because there is a top level >assignment). And almost certainly unnecessary. IME, those are all to easily make bare class definitions new style in Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 13:16:09 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:16:09 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071609.1dcfffa6@freewill> On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: >Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > >That should work in Python 2.X and 3.X. Ugly, but good call! :) >I've got argparse running on Python 2.3-3.1, and the changes were >pretty easy. You can see them all in the revision here: > > http://code.google.com/p/argparse/source/detail?r=12 > >I have aspirations of putting all of the tricks I learned up up on the >Wiki somewhere, but I just haven't had the time. The more resources we can provide people, both in code and in documentation, the better. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Tue Jan 12 13:29:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 12:29:12 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112071609.1dcfffa6@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> Message-ID: <4B4C6B18.7050008@voidspace.org.uk> On 12/01/2010 12:16, Barry Warsaw wrote: > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > >> Actually there's a solution to this one too: >> >> FooBase = Meta('FooBase', (), {}) >> class Foo(FooBase): >> ... >> >> That should work in Python 2.X and 3.X. >> > Ugly, but good call! :) > > There are all sorts of tricks. For example you can do exception handling that works with pre-2.6 syntax and 3.0 with a bare except and using sys.exc_info. It is horrible, but acceptable for short pieces of code (I have a couple of small modules that do this). I haven't yet tried converting larger code-bases to Python 3, but I think the workflow advocated by Martin is greatly preferable to the hacks and tricks needed to make the same codebase run under 2 & 3. Michael >> I've got argparse running on Python 2.3-3.1, and the changes were >> pretty easy. You can see them all in the revision here: >> >> http://code.google.com/p/argparse/source/detail?r=12 >> >> I have aspirations of putting all of the tricks I learned up up on the >> Wiki somewhere, but I just haven't had the time. >> > The more resources we can provide people, both in code and in documentation, > the better. > > Thanks! > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 12 14:09:50 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Jan 2010 14:09:50 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C749E.4040009@egenix.com> Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > * argparse (PEP 389) > * brief mention on still wanting to break out the stdlib from CPython > * an official policy on extension modules? (i.e. must have a pure Python > implementation which can mean a ctypes implementation unless given an > explicit waiver) > > That's everything from a stdlib perspective. I have half-baked ideas, but > nothing concrete enough to bring up unless people really want to discuss > stuff like how to potentially standardize module deprecation warnings, etc. > > In terms of me personally, I do plan to bring up at some point during the > summit these points which don't squarely fit during my time slot: > > * an official unofficial policy on how new proposed features should come to > us (i.e. working code to python-ideas with a shell of a PEP that includes > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > If there is something missing from the stdlib discussion that you think > should be brought up at the summit let me know. And if there is something > here you want to discuss before the summit let me know and I can start a > separate thread on it. Could you please put these things and the results up on the Python wiki ?! We're going to have a language summit at EuroPython this year as well and may want to continue/extend the discussion based on what you're doing at PyCon. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 12 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brian.curtin at gmail.com Tue Jan 12 15:33:06 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 12 Jan 2010 08:33:06 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Tue, Jan 12, 2010 at 04:20, Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : > > > > For example, there are currently over > > 1500 open issues with no stage set, some of which seemingly haven't been > > read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. > > > Would a properly set stage field save issues from > > falling into a black hole? > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Agreed, I didn't mean to place blame on the stage field, I just ran with how I view that field since it was mentioned. When I'm thinking in "code-writing developer mode" (tm), I'm more likely to go to issues which have a stage so I know what I'm going into -- what needs to be worked on. When I'm in project cleanup mode, I go by stageless issues -- what is necessary for this to begin or end work. Maybe others work differently...software projects take all kinds :-) Anyways, sorry for the off-topic. If this is a summit worthy discussion, cool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 17:12:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 11:12:28 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C66A6.2040601@gmail.com> References: <4B4C66A6.2040601@gmail.com> Message-ID: <20100112161228.300EA1D688D@kimball.webabinitio.net> On Tue, 12 Jan 2010 22:10:14 +1000, Nick Coghlan wrote: > There are actually a whole host of reasons issues can stagnate: > - a feature request may seem reasonable (hence it doesn't get rejected > outright), but the right API may not be clear (hence it doesn't get > implemented in the near term) > - a patch may be reviewed and found to have significant defects (or > simply be overreaching the stated goal) but the original patch poster > loses interest after meeting resistance in their ambition to fix > something that is "obviously" broken > - a problem may simply be hard to fix in a backwards compatible way (or > even at all!) I would add: - a patch is reviewed but the reviewer requests a second opinion before committing, which never arrives, and the original reviewer forgets about the issue. I have occasionally observed apparently moribund issues spring to life and get resolved when a triage person does something as simple as updating the releases to which an issue applies. > Ultimately it boils down to Antoine's two categories (lack of interest > and lack of relevant expertise to make a final decision) but there are a > lot of different ways to get into those two buckets. There's a third bucket here: lack of time. Sometimes there is interest and expertise, but no time. Worse, sometimes we end up waiting on the person with the expertise and interest but no time, when someone else with somewhat less expertise but more time should just go ahead and do the commit. (*That* one should be fixable.) > Because we aren't ruthless about pruning those kinds of issues out of > the tracker they're the ones that are going to accumulate over time. Another other category that hangs around in the tracker is items for which there simply isn't a consensus. I suppose those could be brought to python-dev for a final decision if someone wanted to tackle that task. And I don't think it is a lack of ruthlessness. I think it is the current community consensus that we leave these issues open in the tracker in case someone does come along and want to deal with them. (*) > I'd actually be interested to know what the issue stats are like when > RFEs are excluded and when the search is limited to the items flagged as > 'easy'. If easy bug fix issues are taking a long time to get closed than > that would bother me a lot more than RFEs that sit on the tracker for > years. I'd also be interested to know what the average and median lifetime of closed bug reports is. Not the metrics for open issues, but the metrics for closed ones. My theory is that we close a lot of bugs fairly promptly, but the ones in the above categories make the average age of *open* issues high. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls (*) For example, I'm currently going through all the open issues relating to the email package, and some of them are pretty darn old, yet still have useful content. From brett at python.org Tue Jan 12 18:40:13 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:40:13 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C749E.4040009@egenix.com> References: <4B4C749E.4040009@egenix.com> Message-ID: On Tue, Jan 12, 2010 at 05:09, M.-A. Lemburg wrote: > Brett Cannon wrote: > > Michael has given me the hg transition/stdlib time slot at the language > > summit this year. In regards to that I plan to lead a discussion on: > > > > * where we are at w/ the Hg transition (Dirkjan should be there and I did > a > > blog post on this topic recently: > > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > > * argparse (PEP 389) > > * brief mention on still wanting to break out the stdlib from CPython > > * an official policy on extension modules? (i.e. must have a pure Python > > implementation which can mean a ctypes implementation unless given an > > explicit waiver) > > > > That's everything from a stdlib perspective. I have half-baked ideas, but > > nothing concrete enough to bring up unless people really want to discuss > > stuff like how to potentially standardize module deprecation warnings, > etc. > > > > In terms of me personally, I do plan to bring up at some point during the > > summit these points which don't squarely fit during my time slot: > > > > * an official unofficial policy on how new proposed features should come > to > > us (i.e. working code to python-ideas with a shell of a PEP that includes > > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > > * any changes needed to the issue tracker to help with the workflow? > (stage > > field seems like a failed experiment and we now have several effective > > triage people who can help w/ guiding changes) > > > > If there is something missing from the stdlib discussion that you think > > should be brought up at the summit let me know. And if there is something > > here you want to discuss before the summit let me know and I can start a > > separate thread on it. > > Could you please put these things and the results up on the Python > wiki ?! > > We're going to have a language summit at EuroPython this year > as well and may want to continue/extend the discussion based > on what you're doing at PyCon. > > I expect there will be at least summary emails on what gets discussed. There is also a chance that it will be videotaped. -Brett > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 12 2010) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 18:47:50 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:47:50 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? >> (stage field seems like a failed experiment and we now have several >> effective triage people who can help w/ guiding changes) >> >> -Brett >> > I think it would be interesting to see how people are using the tracker, or > how they want to be using it. For example, there are currently over 1500 > open issues with no stage set, some of which seemingly haven't been read by > anyone at all. Would a properly set stage field save issues from falling > into a black hole? > > "Using" is the key word there. I know this thread went on about how bugs tend to end up being left open, but what I was proposing to talk about was whether there is any shift desired by the seasoned tracker users to help in their work. For instance, the Stage field was meant to help, but I don't think it is really getting used much, which makes it at best just a UI junk and at worst something to confuse new users. So I just wanted to discuss things as a group to see if we could streamline some things, add others, etc. At worst I will try to grab people like Mark D., R. David, Antoine, Georg, etc. at the summit (not sure exactly which the heavy bug closers will be there), take them aside, and flat-out ask what they need to make their jobs easier. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 19:29:06 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:29:06 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C6B18.7050008@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> <4B4C6B18.7050008@voidspace.org.uk> Message-ID: On Tue, Jan 12, 2010 at 04:29, Michael Foord wrote: > On 12/01/2010 12:16, Barry Warsaw wrote: > > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > > > Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > > That should work in Python 2.X and 3.X. > > > Ugly, but good call! :) > > > > > There are all sorts of tricks. For example you can do exception handling > that works with pre-2.6 syntax and 3.0 with a bare except and using > sys.exc_info. It is horrible, but acceptable for short pieces of code (I > have a couple of small modules that do this). > > I haven't yet tried converting larger code-bases to Python 3, but I think > the workflow advocated by Martin is greatly preferable to the hacks and > tricks needed to make the same codebase run under 2 & 3. > > In other words we need to pull together a HOWTO for Python source like the one for extension modules that Benjamin wrote and make it rather prominently linked from the Python 3 documentation index page. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 19:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 13:31:23 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? > >> (stage field seems like a failed experiment and we now have several > >> effective triage people who can help w/ guiding changes) > >> > > I think it would be interesting to see how people are using the tracker, or > > how they want to be using it. For example, there are currently over 1500 > > open issues with no stage set, some of which seemingly haven't been read by > > anyone at all. Would a properly set stage field save issues from falling > > into a black hole? > > > "Using" is the key word there. I know this thread went on about how bugs > tend to end up being left open, but what I was proposing to talk about was > whether there is any shift desired by the seasoned tracker users to help in > their work. For instance, the Stage field was meant to help, but I don't > think it is really getting used much, which makes it at best just a UI junk > and at worst something to confuse new users. So I just wanted to discuss > things as a group to see if we could streamline some things, add others, > etc. Well, I for one find the stage field useful. It reminds me at a glance where the issue is at in the workflow (and 'not set' is a valid place in the workflow that an issue sometimes stays in for a while for reason other than not having been triaged yet). I suspect that if we discussed it as a group (we who make the most changes on the tracker) we could improve it, and the interface in general. By the way, you could talk to people who aren't going to be at the summit on #python-dev; I think all the currently tracker-active people hang out there on a regular basis. I'll have to give some thought to what changes/improvements might be most useful, now that I've been doing this for a while. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Tue Jan 12 19:34:02 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:34:02 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> References: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> Message-ID: On Tue, Jan 12, 2010 at 10:31, R. David Murray wrote: > On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > > On Mon, Jan 11, 2010 at 17:57, Brian Curtin > wrote: > > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > > >> * any changes needed to the issue tracker to help with the workflow? > > >> (stage field seems like a failed experiment and we now have several > > >> effective triage people who can help w/ guiding changes) > > >> > > > I think it would be interesting to see how people are using the > tracker, or > > > how they want to be using it. For example, there are currently over > 1500 > > > open issues with no stage set, some of which seemingly haven't been > read by > > > anyone at all. Would a properly set stage field save issues from > falling > > > into a black hole? > > > > > "Using" is the key word there. I know this thread went on about how bugs > > tend to end up being left open, but what I was proposing to talk about > was > > whether there is any shift desired by the seasoned tracker users to help > in > > their work. For instance, the Stage field was meant to help, but I don't > > think it is really getting used much, which makes it at best just a UI > junk > > and at worst something to confuse new users. So I just wanted to discuss > > things as a group to see if we could streamline some things, add others, > > etc. > > Well, I for one find the stage field useful. It reminds me at a glance > where the issue is at in the workflow (and 'not set' is a valid place > in the workflow that an issue sometimes stays in for a while for reason > other than not having been triaged yet). I suspect that if we discussed > it as a group (we who make the most changes on the tracker) we could > improve it, and the interface in general. > > By the way, you could talk to people who aren't going to be at the > summit on #python-dev; I think all the currently tracker-active people > hang out there on a regular basis. > > Sounds reasonable. > I'll have to give some thought to what changes/improvements might be > most useful, now that I've been doing this for a while. How about everyone who is active on bugs.python.org give it some thought and we can try to have a discussion at some point in the relatively near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jan 12 21:58:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 06:58:49 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: <4B4C749E.4040009@egenix.com> Message-ID: <4B4CE289.6040709@gmail.com> Brett Cannon wrote: > I expect there will be at least summary emails on what gets discussed. > There is also a chance that it will be videotaped. The Wiki makes for a better summary archive though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jan 12 22:53:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:53:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <4B4CEF3D.8000107@v.loewis.de> >> a) telling people that they have to move to 2.6 first actually >> hurts migration, instead of helping, because it implies to them >> that they have to drop old versions (e.g. 2.3.) - just because >> they had *always* dropped old versions before supporting new ones. > > Is it just an implication, or is it reality? That's only the implication. However, this was precisely the dialogue when talking to Django. If you start with "start supporting 2.6", the immediate response, without listening further, was, "ok, wait until we drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). Then explain it to the individual you are talking to, wait for the next developer of the project step along, and see how he brings up the very same line of thinking (supporting new versions == dropping support for old versions). I think only part of that comes from the maintenance burden. The other part is that they *want* to drop support for old versions, so that they can eventually start using new features (e.g. generator expressions). So they welcome the requirement to support new versions as an excuse to drop old ones ("it is obvious that you have to drop 2.3 to support 3.2"). However, their users then won't let them drop old versions. > >> b) IMO, people also don't gain much by first migrating to 2.6. >> In principle, it gives them the opportunity to get py3k warnings. >> However, I haven't heard a single positive report where these >> warnings have actually helped people in porting. Yours is the >> first report saying that you followed the official guideline, >> but you didn't state whether doing so actually helped (or whether >> you just ported to 2.6 because the guideline told you to). > > Python 2.6 has other useful features, which I want to take advantage of I think you are a minority with that, being able to actually use the 2.6 features already. Many projects can't, as they have to support at least 2.4 still (so the with statement is right out). >> Inherently, 2.8 can't improve on that. > > I'm not so sure. Yes, as a package maintainer there are older versions to > think about, but time moves on for everyone (hopefully :). By the time 2.8 is > released, what will be the minimum version of Python provided by most OS > vendors (where the majority of Python users probably get their 'python')? "Current" Linux distributions will have 2.6 then. "Old" installations will have 2.4. > I > guess some people will have to support everything from Python 2.3 to 2.8 but > you're talking supporting something like a spread of 7 years of Python > versions. What other platform do you support for 7 years? I think 2.3 will really be gone by the time 2.8 might get released. Even with 2.7, you'd end up with a span of seven years, though. Python had been supporting Windows 95 for more than 7 years (I think rather 9 or 10), likewise Windows 3.1 before that. Python 2.7 will likely still support Windows 2000, which then will be ten years old. Solaris support will probably go back to Solaris 2.6, which will be 13 years old when Python 2.7 gets released. It's only the Linux (and OS X) releases that move so quickly. Regards, Martin From martin at v.loewis.de Tue Jan 12 22:56:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:56:55 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> Message-ID: <4B4CF027.4080007@v.loewis.de> > Maybe not, but the Distribute feature is there because IMO the > distutils feature by itself isn't particularily useful. You need to > write your own distutils extensions, in practice, and they are not > trivial. I wouldn't say that. My Django port works with bare distutils (as does Django itself), and it works fine. That distribute had to redo it is only because setuptools *replaces* the build_py command, as does the 2to3 support in distutils. So only if you have a different build_py already, you can't use what is in distutils. Regards, Martin From fuzzyman at voidspace.org.uk Tue Jan 12 23:02:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:02:34 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CEF3D.8000107@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> Message-ID: <4B4CF17A.1000503@voidspace.org.uk> On 12/01/2010 21:53, "Martin v. L?wis" wrote: >>> a) telling people that they have to move to 2.6 first actually >>> hurts migration, instead of helping, because it implies to them >>> that they have to drop old versions (e.g. 2.3.) - just because >>> they had *always* dropped old versions before supporting new ones. >>> >> Is it just an implication, or is it reality? >> > That's only the implication. However, this was precisely the dialogue > when talking to Django. If you start with "start supporting 2.6", the > immediate response, without listening further, was, "ok, wait until we > drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). > > Then explain it to the individual you are talking to, wait for the next > developer of the project step along, and see how he brings up the > very same line of thinking (supporting new versions == dropping support > for old versions). > > I think only part of that comes from the maintenance burden. The other > part is that they *want* to drop support for old versions, so that they > can eventually start using new features (e.g. generator expressions). > So they welcome the requirement to support new versions as an excuse > to drop old ones ("it is obvious that you have to drop 2.3 to support > 3.2"). However, their users then won't let them drop old versions. > > > I agree with Martin that the *perception* is that to use Python 2.6 to help you port to Python 3 you have to be willing to drop support for earlier versions of Python. >> >>> b) IMO, people also don't gain much by first migrating to 2.6. >>> In principle, it gives them the opportunity to get py3k warnings. >>> However, I haven't heard a single positive report where these >>> warnings have actually helped people in porting. Yours is the >>> first report saying that you followed the official guideline, >>> but you didn't state whether doing so actually helped (or whether >>> you just ported to 2.6 because the guideline told you to). >>> >> Python 2.6 has other useful features, which I want to take advantage of >> > I think you are a minority with that, being able to actually use the 2.6 > features already. Many projects can't, as they have to support at least > 2.4 still (so the with statement is right out). > > Well, the IronPython community has almost completely moved over to IronPython 2.6. :-) We tend to ship our Python runtime with our applications though. As it happens I'm now working with CPython on the server (Python 2.5) and IronPython in the browser where we are using 2.6. The new property feature is nice, as is having with without a __future__ import. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From regebro at gmail.com Tue Jan 12 23:03:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 23:03:41 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF027.4080007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> <4B4CF027.4080007@v.loewis.de> Message-ID: <319e029f1001121403x14ef7ec8x729fb80fa67feaef@mail.gmail.com> On Tue, Jan 12, 2010 at 22:56, "Martin v. L?wis" wrote: >> Maybe not, but the Distribute feature is there because IMO the >> distutils feature by itself isn't particularily useful. You need to >> write your own distutils extensions, in practice, and they are not >> trivial. > > I wouldn't say that. My Django port works with bare distutils (as > does Django itself), and it works fine. > > That distribute had to redo it is only because setuptools *replaces* > the build_py command, as does the 2to3 support in distutils. So only > if you have a different build_py already, you can't use what is in > distutils. Yeah, you are right, I misremembered. The actual additional feature is the support for the test command. Testing under Python 2 and 3 without it is annoying. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 12 23:04:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:04:37 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <4B4CF1F5.4050403@v.loewis.de> > [...] >> I've done a fair bit of 3.x porting, and I'm firmly convinced that >> 2.x can do nothing: > [...] >> Inherently, 2.8 can't improve on that. > > I agree that there are limitations like the ones you've listed, but I > disagree with your conclusion. Maybe you assume that it's just as hard > to move to 2.8 (using the py3k backports not available in say 2.5) as it > is to 3.x? Not at all, no. I'd rather expect that code that runs on 2.7 will run on 2.8 unmodified. > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? > I think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? > Then, when all of your dependencies (or viable alternatives to those > dependencies) are available for 3.x, you'll have an easier transition if > you can start from a 2.x with fewer differences in features. But you won't *have* fewer differences. Just because your code runs on 2.8 doesn't mean it will stop running on 2.3 (if you have a need for that). This doesn't get you any closer - you can't use any of the 2.8 features as long as you have to support older versions of Python. > Fundamentally the more 2.x can converge on 3.x, the easier it will be > for users to make the leap, because it will be a smaller leap. No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. > The > longer the 2.x series lives, the more these newer 2.x versions like 2.7 > and maybe even 2.8 will be available on common platforms for people to > depend upon as minimum versions, which means that as time goes by they > can depend on a version that's closer to 3.x. No, that's incorrect. Suppose 2.7 is the last 2.x release, to be released in 2010. Then, in 2015, it may be that everybody has migrated to 2.7, which then is a common platform. If you release 2.8 in 2012, then, in 2015, people will be split between 2.7 and 2.8, and so there won't be a common platform before 2017. So stopping 2.x development *earlier* will also give us a common platform earlier. Regareds, Martin From martin at v.loewis.de Tue Jan 12 23:07:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:07:02 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4CF286.5040002@v.loewis.de> > I think it would be interesting to see how people are using the tracker, > or how they want to be using it. For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. Would a properly set stage field save issues from > falling into a black hole? I personally don't think this would be the case. What would help is people who actually *work* on the issues, rather than merely reading them. However, this being open source, there is no way to force it (unless you create an incentive, such as the five-for-one deal). Regards, Martin From python at mrabarnett.plus.com Tue Jan 12 23:10:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jan 2010 22:10:28 +0000 Subject: [Python-Dev] regex module Message-ID: <4B4CF354.2050603@mrabarnett.plus.com> Hi all, I'm back on the regex module after doing other things and I'd like your opinion on a number of matters: Firstly, the current re module has a bug whereby it doesn't split on zero-width matches. The BDFL has said that this behaviour should be retained by default in case any existing software depends on it. My question is: should my regex module still do this for Python 3? Speaking personally, I'd like it to behave correctly, and Python 3 is the version where backwards-compatibility is allowed to be broken. Secondly, Python 2 is reaching the end of the line and Python 3 is the future. Should I still release a version that works with Python 2? I'm thinking that it could be confusing if new regex module did zero-width splits correctly in Python 3 but not in Python 2. And also, should I release it only for Python 3 as a 'carrot'? Finally, the module allows some extra backslash escapes, eg \g, in the pattern. Should it treat ill-formed escapes, eg \g, as it would have treated them in the re module? Thanks From michael at voidspace.org.uk Tue Jan 12 23:46:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:46:49 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 Message-ID: <4B4CFBD9.1090009@voidspace.org.uk> I presume the email below is about the Windows binary. Does the AMD64 release work on intel 64bit and can we make the wording clearer on the download page? The current description is " Windows AMD64 binary". All the best, Michael -------- Original Message -------- Subject: Download Page - AMD64 Date: Fri, 11 Dec 2009 04:03:16 -0600 From: Thomas Brownback To: webmaster at python.org Should I use the AMD64 version of Python on an Intel 64 chip? I know those 64-bit implementations are very similar, but are they similar enough that your AMD64 will work on Intel? If so, perhaps a note should be placed on that page to help avoid confusion. Explicit being better than implicit and all. Thanks for your time, Thomas -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at bemusement.org Tue Jan 12 23:49:56 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Wed, 13 Jan 2010 09:49:56 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100112224956.GC28576@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > But a hypothetical 2.8 would also give people a way to move closer to > > py3k without giving up on using all their 2.x-only dependencies. > > How so? If they use anything that is new in 2.8, they *will* need to > drop support for anything before it, no??? > > > I think it's much more likely that libraries like Twisted can support 2.8 > > in the near future than 3.x. > > Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does > that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. From tjreedy at udel.edu Tue Jan 12 23:51:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 17:51:31 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: On 1/12/2010 5:04 PM, "Martin v. L?wis" wrote: > But you won't *have* fewer differences. Just because your code runs > on 2.8 doesn't mean it will stop running on 2.3 (if you have a need > for that). This doesn't get you any closer - you can't use any of > the 2.8 features as long as you have to support older versions of > Python. > >> Fundamentally the more 2.x can converge on 3.x, the easier it will be >> for users to make the leap, because it will be a smaller leap. > > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. > >> The >> longer the 2.x series lives, the more these newer 2.x versions like 2.7 >> and maybe even 2.8 will be available on common platforms for people to >> depend upon as minimum versions, which means that as time goes by they >> can depend on a version that's closer to 3.x. > > No, that's incorrect. Suppose 2.7 is the last 2.x release, to be > released in 2010. Then, in 2015, it may be that everybody has migrated > to 2.7, which then is a common platform. > > If you release 2.8 in 2012, then, in 2015, people will be split between > 2.7 and 2.8, and so there won't be a common platform before 2017. Just like people today may need to work with both 2.5 and 2.6, or privately backport 2.6 bugfixes to 2.5. > So stopping 2.x development *earlier* will also give us a common > platform earlier. With years of bug fixes and hence high quality. From tjreedy at udel.edu Wed Jan 13 00:05:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 18:05:12 -0500 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On 1/12/2010 5:10 PM, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. Are you writing a new module with a new name? If so, do you expect it to replace or augment re? (This is the same question as for optparse vs. argparse, which I understand to not yet be decided.) > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, 2.7 stdlib is already out. A new engine should get some community testing before going in the stdlib. Even 3.2 beta is not that far off (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? What does re do with analogous cases? Terry Jan Reedy From david.lyon at preisshare.net Wed Jan 13 00:20:54 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 13 Jan 2010 10:20:54 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C4589.70906@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> <4B4C4589.70906@gmail.com> Message-ID: <1333.218.214.45.58.1263338454.squirrel@syd-srv02.ezyreg.com> > Nick wrote: >>> This has nothing to do with pushing 3.x, but all with managing >>> available manpower and still providing quality software. >> >> Python 3.x needs more carrots. > > As Guido has said a few times, the gains are far greater for *new* > Python developers than they are for existing ones. Well both you and Guido are most likely 100% correct. :-) > They don't have as rich a 3rd party ecosystem > on Python 3 as they would on Python 2.x at this point in time, but > unlike existing developers they also have the luxury of cherry-picking > just those packages that already have Python 3 support. Most likely. I wouldn't want to say anything to discourage people from going to python 3. In other languages, I have much experience of making the jump to 'a whole new world'. It's unsettling at first, but after that, you suddenly find the new model has a better turbo than the last and you're at the next corner faster than you can think. So it's all good. But I still maintain Python 3.0 needs more carrots. For example, if mercurial or any other cool lib gets added to python 3 (and I can name a few that should go in python 3) then they should be added to python 3 and not to python 2.x They would serve as good carrots. Make fresh the python 3 stdlib and preserve the python 2.x stdlib. I really think we are somewhat short on resources to do what Guido has asked about bringing python up to CPAN level with respect to packages. We're starting a long way back with horses and swords and trying to catch a well fed and greased modern machine.. I'm sure we can get a modern package testbot operational for python. But I wish those who were complaining about the packaging problem so much could throw some money at the problem instead of moaning about it. As is done on other open-source projects. Some organisation would be beneficial. Finding funds so that a small group of people could work on the problem would be a great boost to forward progress. I just think python packaging needs six months of that to repair the years and years of neglect and stagnation. Even if it is only beer and bus fare money. It just needs a temporary shot of adrenalin in the arm.. so to speak.. Regards David From martin at v.loewis.de Wed Jan 13 00:28:14 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 00:28:14 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D058E.4030404@v.loewis.de> > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? "intel 64bit" is as clear as mud. It could mean the "Intel 64" architecture, or it could mean the "IA-64" architecture, both are 64-bit architectures with processors manufactured by Intel. The former is officially called AMD64 (not by Intel, but officially), and our AMD64 binaries run on such processors. The latter is also called "Itanium", and we stopped producing binaries for that architecture a while ago; the AMD64 binaries will *not* run on it. The wording could probably be changed to match the reader's expectation more, most likely, it would also get more incorrect in the process. To make it correct and explicit, an entire paragraph educating users about 64-bit architectures and that there are many of them and that they are mutually incompatible might be required. Most likely, Thomas' processor implements the AMD64 architecture, even though it is manufactured by Intel. A short sentence that he would probably understand (given that he used "Intel 64", not "64bit") is: """The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).""" Regards, Martin From martin at v.loewis.de Wed Jan 13 00:30:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 00:30:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112224956.GC28576@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> <20100112224956.GC28576@steerpike.home.puzzling.org> Message-ID: <4B4D0613.1010503@v.loewis.de> > I'm not talking about Twisted moving to 3.x (FWIW, I think the only > movement there so far is some patches for some -3 warnings). The > situation I'm describing is a project X that: > > (a) has 2.x-only dependencies, and > (b) would like to be as close as possible to 3.x (because they like > the new features and/or want to be as ready as possible to jump > when (a) is fixed). This assumes that jumping to 3.x is easier if you are closer to it. Please trust me that this is not the case. >> No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they >> likely won't. > > But this is my point. I think they would as an intermediate step to > jumping to 3.x (which also requires dropping 2.5, after all!), if for > some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. No, and no. No: it's not an intermediate step, and no, supporting 3.x does *not* require dropping 2.5. Regards, Martin From fuzzyman at voidspace.org.uk Wed Jan 13 00:40:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:40:35 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D058E.4030404@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> Message-ID: <4B4D0873.5070006@voidspace.org.uk> On 12/01/2010 23:28, "Martin v. L?wis" wrote: > [snip...] > """The binaries for AMD64 will also work on processors that implement > the Intel 64 architecture (formerly EM64T), i.e. the architecture that > Microsoft calls x64, and AMD called x86-64 before calling it AMD64. > They will not work on Intel Itanium Processors (formerly IA-64).""" > > To those not intimately aware of the history and present of 64 bit architecture this sentence would be a great addition - thanks. I'm adding it now as a footnote. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From lists at cheimes.de Wed Jan 13 00:41:04 2010 From: lists at cheimes.de (Christian Heimes) Date: Wed, 13 Jan 2010 00:41:04 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D0890.2030505@cheimes.de> Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". The installer works on all AMD64 compatible Intel CPUs. *scnr* As you most likely know all modern Intel 64bit CPUs are based on AMD's x86-64 design. Only the Itanium family is based on the other Intel 64bit design: IA-64. The name AMD64 was chosen because most Linux and BSD distributions call it so. The name 'AMD64' has caused confusion in the past because users thought that the installer works only on AMD CPUs. How about: * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / X86-64 binary -- does not include source) instead of: * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does not include source) ? Christia From fuzzyman at voidspace.org.uk Wed Jan 13 00:55:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:55:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0873.5070006@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> <4B4D0873.5070006@voidspace.org.uk> Message-ID: <4B4D0BD4.5050109@voidspace.org.uk> On 12/01/2010 23:40, Michael Foord wrote: > On 12/01/2010 23:28, "Martin v. L?wis" wrote: >> [snip...] >> """The binaries for AMD64 will also work on processors that implement >> the Intel 64 architecture (formerly EM64T), i.e. the architecture that >> Microsoft calls x64, and AMD called x86-64 before calling it AMD64. >> They will not work on Intel Itanium Processors (formerly IA-64).""" >> > > To those not intimately aware of the history and present of 64 bit > architecture this sentence would be a great addition - thanks. I'm > adding it now as a footnote. > I can add it now to the main download page and existing release pages - but are there changes needed to any scripts to change pages created for *new* releases? All the best, Michael Foord > All the best, > > Michael Foord > >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From python at mrabarnett.plus.com Wed Jan 13 01:22:01 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 00:22:01 +0000 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D1229.70708@mrabarnett.plus.com> Terry Reedy wrote: > On 1/12/2010 5:10 PM, MRAB wrote: >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. > > Are you writing a new module with a new name? If so, do you expect it to > replace or augment re? (This is the same question as for optparse vs. > argparse, which I understand to not yet be decided.) > It's a module called 'regex'. It can be used in place of 're' by using "import regex as re", except for differences such as "\g" being a legal group reference in pattern strings. >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? > > 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, > 2.7 stdlib is already out. A new engine should get some community > testing before going in the stdlib. Even 3.2 beta is not that far off > (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? > > What does re do with analogous cases? > The 're' module treats r"\g" as "g"; both 're' and 'regex' treat, say, r"\q" as "q". The closest analogue to what I'm asking about is that re treats the ill-formed repeat r"x{1," as a literal, which sort of suggests that r"\g" should be treated as "g", but r"\g" is now a group reference (re would treat that as "g". Does that sound reasonable? From fuzzyman at voidspace.org.uk Wed Jan 13 01:50:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 00:50:39 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D18DF.6070502@voidspace.org.uk> On 12/01/2010 23:41, Christian Heimes wrote: > Michael Foord wrote: > >> I presume the email below is about the Windows binary. Does the AMD64 >> release work on intel 64bit and can we make the wording clearer on the >> download page? >> >> The current description is " Windows AMD64 binary". >> > The installer works on all AMD64 compatible Intel CPUs. *scnr* > > As you most likely know all modern Intel 64bit CPUs are based on AMD's > x86-64 design. Only the Itanium family is based on the other Intel 64bit > design: IA-64. The name AMD64 was chosen because most Linux and BSD > distributions call it so. The name 'AMD64' has caused confusion in the > past because users thought that the installer works only on AMD CPUs. > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) > Right - I've made that change for the Python 2.6, 2.7, 3.0 and 3.1 download pages with a footnote. Prior to that we were offering ia64 *and* amd64 releases so I've left those unchanged. Thanks, Michael > ? > > Christia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From exarkun at twistedmatrix.com Wed Jan 13 02:40:03 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 13 Jan 2010 01:40:03 -0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100113014003.1898.1305834328.divmod.xquotient.219@localhost.localdomain> On 12 Jan, 10:04 pm, martin at v.loewis.de wrote: >>[...] >>>I've done a fair bit of 3.x porting, and I'm firmly convinced that >>>2.x can do nothing: >>[...] >>>Inherently, 2.8 can't improve on that. >> >>I agree that there are limitations like the ones you've listed, but I >>disagree with your conclusion. Maybe you assume that it's just as >>hard >>to move to 2.8 (using the py3k backports not available in say 2.5) as >>it >>is to 3.x? > >Not at all, no. I'd rather expect that code that runs on 2.7 will run >on 2.8 unmodified. >>But a hypothetical 2.8 would also give people a way to move closer to >>py3k without giving up on using all their 2.x-only dependencies. > >How so? If they use anything that is new in 2.8, they *will* need to >drop support for anything before it, no??? >>I think it's much more likely that libraries like Twisted can support >>2.8 >>in the near future than 3.x. > >Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does >that help Twisted in moving to 3.2? I'm not reading this thread carefully enough to make any arguments on either side, but I can contribute a fact. Twisted very likely does not support 2.8 today. I base this on the fact that Twisted does not support 2.7 today, and I expect 2.8 will be more like 2.7 than it will be like 2.3 - 2.6 (which Twisted does support). When I say "support" here, I mean "all of the Twisted unit tests pass on it". Jean-Paul From tseaver at palladion.com Wed Jan 13 03:57:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jan 2010 21:57:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4D36AA.7020607@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karen Tracey wrote: > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> >> > Well yay. Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference. That's > down from 104 failures and 40 errors with 2.7 alpha 1. > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Just to add another "success" data point: Zope2's trunk, as well as the 2.12 release, passes all tests (2535 on the trunk) and brings up the appserver just fine under 2.7a2. There is an obnoxious deprecation warning out of the distutils: DeprecationWarning: 'compiler' specifies the compiler type in build_ext. If you want to get the compiler object itself, use 'compiler_obj' which is likely a simple one-line fix, if I only knew what the hell it was whining about. ;) The warning is extra obnoxious because it doesn't tell me what in *my* code triggers the warning (it (needs a 'stacklevel' argument). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktNNqQACgkQ+gerLs4ltQ7d5gCcCGKVjyOlxnrAln0UnRibS7kd lNIAoIs1RlSGMtJWaY11BqptfDmQvR87 =mIOO -----END PGP SIGNATURE----- From python at mrabarnett.plus.com Wed Jan 13 04:09:34 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 03:09:34 +0000 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D396E.4050500@mrabarnett.plus.com> MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > I've just noticed something odd about the re module: the sub() method doesn't take 'pos' or 'endpos' arguments. search() does; match() does; findall() does(); finditer() does; but sub() doesn't. Maybe there has never been a demand for it. (Nor split(), for that matter.) From brett at python.org Wed Jan 13 04:58:11 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 19:58:11 -0800 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On Tue, Jan 12, 2010 at 14:10, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > If it is a separate module under a different name it can do the proper thing. People will just need to be aware of the difference when they import the module. > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > That's totally up to you. There is practically no chance of it getting into the 2.x under the stdlib at this point since 2.7b1 is coming up and this module has not been out in the wild for a year (to my knowledge). If you want to support 2.x that's fine and I am sure users would appreciate it, but it isn't necessary to get into the Python 3 stdlib. > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > > If you want to minimize the differences then it should probably match. As I said, since it is a different name to import under it can deviate where reasonable, just make sure to clearly document the deviations. -Brett > Thanks > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jan 13 07:11:49 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 07:11:49 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D6425.3060307@v.loewis.de> > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) -1. AMD doesn't want us to use the term x86-64 anymore, but wants us to use AMD64 instead. I think we should comply - they invented the architecture, so they have the right to give it a name. Neither Microsoft nor Intel have such a right. Regards, Martin From sridharr at activestate.com Wed Jan 13 07:47:38 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 12 Jan 2010 22:47:38 -0800 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D6C8A.7070307@activestate.com> On 1/12/2010 2:46 PM, Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". FWIW, we simply use (64-bit, x64). Platform Download ============================================== Windows (x86) Windows Installer (MSI) Windows (64-bit, x64) Windows Installer (MSI) https://www.activestate.com/activepython/downloads/ -srid From solipsis at pitrou.net Wed Jan 13 08:08:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Jan 2010 07:08:55 +0000 (UTC) Subject: [Python-Dev] Fwd: Download Page - AMD64 References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) +1. I don't care about trademarks or official names, we should call it whatever is obvious for our users. As for Itanium, it is practically dead, I think there is little chance of people mixing it up with x86-64. cheers Antoine. From michael at voidspace.org.uk Wed Jan 13 10:32:00 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 09:32:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4D9310.6060907@voidspace.org.uk> On 13/01/2010 06:11, "Martin v. L?wis" wrote: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) >> >> instead of: >> >> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >> not include source) >> > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. > I think we should use whatever is most informative and least confusing to our users - we owe our allegiance to them and not to a processor vendor. Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Wed Jan 13 11:30:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:30:50 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF17A.1000503@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> Message-ID: <4B4DA0DA.7070007@gmail.com> Michael Foord wrote: > I agree with Martin that the *perception* is that to use Python 2.6 to > help you port to Python 3 you have to be willing to drop support for > earlier versions of Python. Note that increased 3.x compatibility in the most recent 2.x release will always help in two scenarios: 1. New projects that want to use 2.x only libraries but want to be ready for the Py3k transition in their own code (e.g. new 2.7 features like set literals, dict and set comprehensions and the view* dict methods can be translated far more cleanly by 2to3 than the closest comparable 2.6 code). 2. Similarly new projects that use a 3.x trunk can be more readily translated to a 2.7 version with 3to2, whereas some constructs may be difficult to recreate in earlier Python versions. I would expect this to be significantly less common then the first scenario though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:38:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:38:31 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4DA2A7.2080408@gmail.com> Antoine Pitrou wrote: > Christian Heimes cheimes.de> writes: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) > > +1. I don't care about trademarks or official names, we should call it whatever > is obvious for our users. Until this discussion, I didn't even know the architecture had any name other than x86-64. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:39:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:39:40 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4D36AA.7020607@palladion.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> Message-ID: <4B4DA2EC.3050908@gmail.com> Tres Seaver wrote: > There is an obnoxious deprecation warning out of the distutils: > > DeprecationWarning: 'compiler' specifies the compiler type in > build_ext. If you want to get the compiler object itself, use > 'compiler_obj' > > which is likely a simple one-line fix, if I only knew what the hell it > was whining about. ;) The warning is extra obnoxious because it doesn't > tell me what in *my* code triggers the warning (it (needs a 'stacklevel' > argument). Could you kick a tracker issues in Tarek's direction for that one? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Wed Jan 13 13:24:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 13 Jan 2010 12:24:09 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Brett Cannon python.org> writes: > If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. I'm sorry I won't be able to come to the language summit, but I would like if possible to expedite a pronouncement on PEP 391 (configuring logging using dictionaries). I believe I addressed all the comments made on the discussion threads mentioned in the PEP and so I'm not sure what more I need to do to get a pronouncement. I guess the stdlib slot gives an opportunity for people to air their views and so I'd be grateful if you added it to the agenda. Thanks & regards, Vinay Sajip From murman at gmail.com Wed Jan 13 14:35:51 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 07:35:51 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. I see and agree with the motivation behind your point, but it's just as reasonable to mimic the name Microsoft uses: the release is for Windows rather than the processor. On MSDN Microsoft uses parentheticals x86, ia64 and x64; this would suggest a name like Python 2.6.4 installer for Windows (x64). Unfortunately this usage doesn't seem to be reflected in consumer-oriented product pages, so I'm uncertain how clear it is for those downloading Python. -- Michael Urman From ncoghlan at gmail.com Wed Jan 13 15:04:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 00:04:28 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4DD2EC.3030709@gmail.com> Michael Urman wrote: > On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. > > I see and agree with the motivation behind your point, but it's just > as reasonable to mimic the name Microsoft uses: the release is for > Windows rather than the processor. On MSDN Microsoft uses > parentheticals x86, ia64 and x64; this would suggest a name like > Python 2.6.4 installer for Windows (x64). Unfortunately this usage > doesn't seem to be reflected in consumer-oriented product pages, so > I'm uncertain how clear it is for those downloading Python. As Windows doesn't run on non-x86 architectures, their naming is generally just Windows (32 bit) and Windows (64 bit). Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. In this case, making it clear that the 64-bit Windows binaries work for both Intel and AMD chips is more important than using only the official architecture name. Cheers, Nick. P.S. This discussion made me realise that out of habit I had dropped the 32-bit version of Python on my new 64-bit Windows box... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Wed Jan 13 17:49:55 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 13 Jan 2010 11:49:55 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA2EC.3050908@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> <4B4DA2EC.3050908@gmail.com> Message-ID: <4B4DF9B3.4030403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Tres Seaver wrote: >> There is an obnoxious deprecation warning out of the distutils: >> >> DeprecationWarning: 'compiler' specifies the compiler type in >> build_ext. If you want to get the compiler object itself, use >> 'compiler_obj' >> >> which is likely a simple one-line fix, if I only knew what the hell it >> was whining about. ;) The warning is extra obnoxious because it doesn't >> tell me what in *my* code triggers the warning (it (needs a 'stacklevel' >> argument). > > Could you kick a tracker issues in Tarek's direction for that one? Done: http://bugs.python.org/issue7694 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktN+bMACgkQ+gerLs4ltQ6s4gCdENhtVQWzoH449BCDz8SNx/4s DEoAn19LMcMs3b6ctdNF8K2hYd6d+BSZ =TWit -----END PGP SIGNATURE----- From rdmurray at bitdance.com Wed Jan 13 18:24:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Jan 2010 12:24:42 -0500 Subject: [Python-Dev] PYTHON3PATH Message-ID: <20100113172442.4A7771FA679@kimball.webabinitio.net> Please review issue 2375 [1], which is an enhancement request to add a PYTHON3PATH environment variable. Because we have elected to have both a python and a python3 command, I think this is an issue worth thinking about carefully to make sure we are serving the Python user community and easing the transition to python3. It could be that "use virtualenv" is the best answer, but I feel we should think about it carefully to make sure that is really true. [1] http://bugs.python.org/issue2375 -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From guido at python.org Wed Jan 13 18:31:39 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:31:39 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: Somehow the bug site doesn't load for me right now, but I'm -1 on this. There are maybe a dozen PYTHON... variables -- we really shouldn't try to add PYTHON3 variants for all of them. Specifically for PYTHONPATH, I feel that its use is always a short-time or localized hack, not something you set in your .profile nd forget about it (forgetting about it inparticular often leads to unnecessary debugging pain later). The better approaches are based on site-packages, wrapper scripts, virtualenv, etc. --Guido On Wed, Jan 13, 2010 at 9:24 AM, R. David Murray wrote: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. ?Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. ?It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 > > -- > R. David Murray ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dirkjan at ochtman.nl Wed Jan 13 18:38:26 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 13 Jan 2010 18:38:26 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 18:29, Benjamin Peterson wrote: > Please consider trying Python 2.7 with your code and reporting any bugs you may I ran the Mercurial test suite on current trunk, and it worked alright. There was a small issue with the fact that mimetypes got fooled by the lack of ImportError from _winreg (which I solved by blacklisting _winreg in demandimport), and one test fails likely because of a change in MIME handling. Will look into that. So the state of trunk looks rather solid from where I sit. Cheers, Dirkjan From ralf at brainbot.com Wed Jan 13 18:40:04 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:40:04 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> (R. David Murray's message of "Wed, 13 Jan 2010 12:24:42 -0500") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87r5ptdhu3.fsf@muni.brainbot.com> "R. David Murray" writes: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 The first thing I got while trying to run a python3 prompt few days ago, was an error. python3 tried to read my $PYTHONSTARTUP file, which used print statements. people will have to run both python 2 and python 3 code at the same time. Using different environment variables will make this easier. - Ralf From ralf at brainbot.com Wed Jan 13 18:55:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:55:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Guido van Rossum's message of "Wed, 13 Jan 2010 09:31:39 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87k4vldh4c.fsf@muni.brainbot.com> Guido van Rossum writes: > Somehow the bug site doesn't load for me right now, but I'm -1 on > this. There are maybe a dozen PYTHON... variables -- we really > shouldn't try to add PYTHON3 variants for all of them. > > Specifically for PYTHONPATH, I feel that its use is always a > short-time or localized hack, not something you set in your .profile > nd forget about it (forgetting about it inparticular often leads to > unnecessary debugging pain later). The better approaches are based on > site-packages, wrapper scripts, virtualenv, etc. then at least remove them completely from python3, so one can use them for his python 2 installation. Otherwise it will stump on some innocent python 3 program (and cause unnecessary debugging pain). From steven.bethard at gmail.com Wed Jan 13 18:57:29 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 13 Jan 2010 09:57:29 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: On Wed, Jan 13, 2010 at 9:40 AM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. ?Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. ?It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. How complicated is your PYTHONSTARTUP file? My suspicion is that you could easily write it to work for both Python 2.X and 3.X. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ssteinerx at gmail.com Wed Jan 13 18:57:42 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 13 Jan 2010 12:57:42 -0500 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> On Jan 13, 2010, at 12:40 PM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. Python 2 will continue to run as it has, and better solutions will emerge for Python 3 development. Beats the heck out of making a whole duplicate set for PYTHON3BLAHBLAH, yuck! S From guido at python.org Wed Jan 13 18:58:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:58:04 -0800 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: Memories of days past... Python had several regular expression implementations before, one of which was called "regex". But I would rather not have a new module -- I would much rather have a flag specifying the new (backwards incompatible) syntax/semantics. The flag would have a long name (e.g. re.NEW_SYNTAX), a short name (e.g. re.N) and an inline syntax, "(?n)...". --Guido On Tue, Jan 12, 2010 at 7:58 PM, Brett Cannon wrote: > > > On Tue, Jan 12, 2010 at 14:10, MRAB wrote: >> >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. >> > > If it is a separate module under a different name it can do the proper > thing. People will just need to be aware of the difference when they import > the module. > >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? >> > > That's totally up to you. There is practically no chance of it getting into > the 2.x under the stdlib at this point since 2.7b1 is coming up and this > module has not been out in the wild for a year (to my knowledge). ?If you > want to support 2.x that's fine and I am sure users would appreciate it, but > it isn't necessary to get into the Python 3 stdlib. > >> >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? >> > > If you want to minimize the differences then it should probably match. As I > said, since it is a different name to import under it can deviate where > reasonable, just make sure to clearly document the deviations. > -Brett > >> >> Thanks >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From ralf at brainbot.com Wed Jan 13 19:03:08 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 19:03:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Steven Bethard's message of "Wed, 13 Jan 2010 09:57:29 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <87fx69dgrn.fsf@muni.brainbot.com> Steven Bethard writes: > > How complicated is your PYTHONSTARTUP file? My suspicion is that you > could easily write it to work for both Python 2.X and 3.X. sure. that's exactly what I did. My point is that sharing those environment variables will cause pain for some people. From guido at python.org Wed Jan 13 19:07:58 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:07:58 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. -1. They have their use, but more in controlled situations. If you have "global" env vars that you only want to use with Python 2.x, write a wrapper for Python 3 that invokes it with -E. -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Wed Jan 13 19:14:21 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 13 Jan 2010 13:14:21 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E0D7D.3040806@scottdial.com> On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Wed Jan 13 19:22:33 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:22:33 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Wed, Jan 13, 2010 at 4:24 AM, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> If there is something missing from the stdlib discussion that you think should > be brought up at the summit let me know. And if there is something here you want > to discuss before the summit let me know and I can start a separate thread on it. > > I'm sorry I won't be able to come to the language summit, but I would like if > possible to expedite a pronouncement on PEP 391 (configuring logging using > dictionaries). I believe I addressed all the comments made on the discussion > threads mentioned in the PEP and so I'm not sure what more I need to do to get a > pronouncement. I guess the stdlib slot gives an opportunity for people to air > their views and so I'd be grateful if you added it to the agenda. Is the PEP in the stage of consensus yet? If so it may not even be necessary to discuss it at the summit (though I certainly won't stop people if they want to). -- --Guido van Rossum (python.org/~guido) From phd at phd.pp.ru Wed Jan 13 19:18:50 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 21:18:50 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <20100113181850.GA3837@phd.pp.ru> On Wed, Jan 13, 2010 at 12:57:42PM -0500, ssteinerX at gmail.com wrote: > Or, how about just removing the antiquated use of environment variables "antiquated"? You are kidding, aren't you?! Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Jan 13 19:51:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:51:14 -0800 Subject: [Python-Dev] PyCon Keynote Message-ID: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Jan 13 20:13:58 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:13:58 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D9310.6060907@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> Message-ID: <4B4E1B76.4010309@v.loewis.de> >>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>> X86-64 binary -- does not include source) >>> >>> instead of: >>> >>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>> not include source) >>> >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. >> > > I think we should use whatever is most informative and least confusing > to our users - we owe our allegiance to them and not to a processor vendor. And why do you think this is x86-64? Regards, Martin From martin at v.loewis.de Wed Jan 13 20:33:24 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:33:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA0DA.7070007@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> <4B4DA0DA.7070007@gmail.com> Message-ID: <4B4E2004.8050905@v.loewis.de> > Note that increased 3.x compatibility in the most recent 2.x release > will always help in two scenarios: > > 1. New projects that want to use 2.x only libraries but want to be ready > for the Py3k transition in their own code (e.g. new 2.7 features like > set literals, dict and set comprehensions and the view* dict methods can > be translated far more cleanly by 2to3 than the closest comparable 2.6 > code). Of these, I can only see this being the case of the view* methods. Why would set literals allow for code that more cleanly translates to 3.x? Suppose you write f = set((1,2,3)) in 2.6. That code will work *exactly* the same way in 3.1, no translation needed at all. Likewise for dict and set comprehensions: the code is as cleanly translated in the 2.7 form as it is in any prior form (ie. no modifications). As for view* methods: there is one important exception where the 2.6 equivalent is as cleanly translated as a view method: for key in D: action This is a more modern equivalent for iterating over D.keys(), so if your code still does the latter, just change it to the former (requires Python 2.2). I'd claim that this is the dominant case of traversing a dictionary (probably followed by iterating over .items()). So while it is true that only view* can be transformed correctly, I'd claim that this is an infrequent issue. > 2. Similarly new projects that use a 3.x trunk can be more readily > translated to a 2.7 version with 3to2, whereas some constructs may be > difficult to recreate in earlier Python versions. That may be true, but is besides the point here: the issue was whether a 2.8 release might help in migrating to 3.x. People who follow this approach already have 3.x code. Whether they would rather port to 2.8 only, and get that work with little effort, or whether they would port to 2.5 and later, and put in more work, I don't know - I guess we would have to ask these people. I would expect that they prefer if 2.x died rather sooner than later, because they can then stop supporting it. Regards, Martin From tjreedy at udel.edu Wed Jan 13 20:36:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Jan 2010 14:36:03 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: On 1/13/2010 2:13 PM, "Martin v. L?wis" wrote: >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. > > And why do you think this is x86-64? I do not believe I have personally seen, or at least noticed, that specific term before. Something like "Release for 64-bit Windows (Win NT, 2000, XP, Vista, 7 ) on AMD64-type processors from AMD and Intel" should be clear enough. From martin at v.loewis.de Wed Jan 13 20:40:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 20:40:30 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E21AE.40602@v.loewis.de> > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). Windows actually does - it runs on IA-64 (which is a non-x86 architecture). However, end users are unlikely to use such hardware, so distinguishing between 32-bit and 64-bit is typically good enough. > In this case, making it clear that the 64-bit Windows binaries work for > both Intel and AMD chips is more important than using only the official > architecture name. Well, we did have Itanium binaries at one point, and the "64-bit Windows binaries" you are referring to most definitely don't run on Itanium, even though that's an Intel chip. Regards, Martin From martin at v.loewis.de Wed Jan 13 20:45:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:45:40 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E0D7D.3040806@scottdial.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> Message-ID: <4B4E22E4.4040504@v.loewis.de> > So to echo what Michael said, the Microsoft nomenclature is "x64" > regardless of yours and Martin's objections to that name. Nobody who > uses Windows would be confused by "x64" since that is *the* Microsoft > naming scheme. That's actually not entirely true. There are several places in the APIs where Microsoft either allows or requires to call the architecture AMD64 (e.g. architecture indication in MSI files). Regards, Martin From regebro at gmail.com Wed Jan 13 20:50:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 20:50:59 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. What do you need to do in the PYTHONSTARTUP file? Ten years of Python programming, and I didn't even know it existed. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From phd at phd.pp.ru Wed Jan 13 21:08:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 23:08:40 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <20100113200840.GC14858@phd.pp.ru> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From murman at gmail.com Wed Jan 13 21:12:11 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 14:12:11 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E22E4.4040504@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> <4B4E22E4.4040504@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 13:45, "Martin v. L?wis" wrote: >> So to echo what Michael said, the Microsoft nomenclature is "x64" >> regardless of yours and Martin's objections to that name. Nobody who >> uses Windows would be confused by "x64" since that is *the* Microsoft >> naming scheme. > > That's actually not entirely true. There are several places in the > APIs where Microsoft either allows or requires to call the architecture > AMD64 (e.g. architecture indication in MSI files). I should have clarified I was talking about the names shown on MSDN subscriptions for downloading installation media of Windows 7 and Windows Vista. It's pretty clear in that context Microsoft uses x64 to describe this platform of Windows. But again, it's far from clear that this is a term they use for non-developers. -- Michael Urman From regebro at gmail.com Wed Jan 13 21:27:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 21:27:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113200840.GC14858@phd.pp.ru> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> Message-ID: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: > On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >> What do you need to do in the PYTHONSTARTUP file? >> Ten years of Python programming, and I didn't even know it existed. :-) > > ? See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Cheese and fries! :-) Anyway, I don't see how this could pose any problems, because any user advanced enough to do these things will be advanced enough to understand what the problem is and fix it so it works under Python 3 too. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ralf at brainbot.com Wed Jan 13 21:52:34 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 20:52:34 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> (Lennart Regebro's message of "Wed, 13 Jan 2010 20:50:59 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <874ompg225.fsf@brainbot.com> Lennart Regebro writes: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) hehe. tab completion: # -*- mode: python -*- # Last changed: 2009-12-23 22:25:15 by ralf import sys import os def initreadline(): try: import readline except ImportError: sys.stdout.write("Module readline not available.\n") return import rlcompleter readline.parse_and_bind("tab: complete") # Use tab for completions readline.parse_and_bind('tab: complete') # This forces readline to automatically print the above list when tab # completion is set to 'complete'. readline.parse_and_bind('set show-all-if-ambiguous on') # Bindings for incremental searches in the history. These searches # use the string typed so far on the command line and search # anything in the previous input history containing them. readline.parse_and_bind('"\C-r": reverse-search-history') readline.parse_and_bind('"\C-s": forward-search-history') history = os.path.expanduser("~/.pyhistory") if os.path.exists(history): readline.read_history_file(history) import atexit atexit.register(lambda: readline.write_history_file(history)) initreadline() del initreadline From martin at v.loewis.de Wed Jan 13 22:05:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 22:05:03 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <4B4E357F.4050107@v.loewis.de> Lennart Regebro wrote: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? I personally set up readline: set tab completion, and load the history of commands from the previous session. Of course, I don't know what Ralf uses it for. Regards, Martin From ncoghlan at gmail.com Wed Jan 13 22:43:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 07:43:41 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <4B4E3E8D.2010407@gmail.com> Guido van Rossum wrote: > On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about > just removing the antiquated use of environment variables altogether > from Python 3 and avoid the issue completely. > > -1. They have their use, but more in controlled situations. If you > have "global" env vars that you only want to use with Python 2.x, > write a wrapper for Python 3 that invokes it with -E. Perhaps a case can be made for Python 3 to assume -E by default, with a -e option to enable reading of the environment variables? That way naive users could run Python3 without tripping over existing Python2 environment variables, while other tools could readily set up a different environment before launching Python3. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Wed Jan 13 23:45:57 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 14:45:57 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 1:43 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users using environment variables? That's a recipe for disaster in any version! :-) -- --Guido van Rossum (python.org/~guido) From jared.grubb at gmail.com Wed Jan 13 23:56:37 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Wed, 13 Jan 2010 14:56:37 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <13F6C43A-D62A-4A9F-AF7A-9BDAC94F7D72@gmail.com> On 13 Jan 2010, at 13:43, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. I raised a question about these PYTHON3* variables once before in a discussion about shebang lines: http://www.mailinglistarchive.com/python-dev at python.org/msg29500.html I'm not advocating them, but just wanted to make sure to bring up the shebang use case. Jared From skip at pobox.com Wed Jan 13 23:24:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 16:24:13 -0600 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <19278.18445.428716.321365@montanaro.dyndns.org> Lennart> What do you need to do in the PYTHONSTARTUP file? Just reading off stuff from my own personal startup file... I use it for stuff I want available during interactive sessions: 1. Enable true division. 2. Conditionally define "help" from back in the days when there was no help builtin function. 3. Auto-save session (readline) history so I can easily recall commands across sessions. 4. Add other fake builtins ("like") or override behavior of some (like "dir") making them handier for interactive use. 5. autoload modules/symbols (pokes around in common modules from sys.excepthook function). Oh, and I've had no particular trouble keeping it working in Python 1, 2 or 3. Skip From fuzzyman at voidspace.org.uk Thu Jan 14 01:20:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 00:20:21 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: <4B4E6345.5070202@voidspace.org.uk> On 13/01/2010 19:13, "Martin v. L?wis" wrote: >>>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>>> X86-64 binary -- does not include source) >>>> >>>> instead of: >>>> >>>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>>> not include source) >>>> >>>> >>> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >>> to use AMD64 instead. I think we should comply - they invented the >>> architecture, so they have the right to give it a name. Neither >>> Microsoft nor Intel have such a right. >>> >>> >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. >> > And why do you think this is x86-64? > Well anecdotal everyone I have *every* talked to about 64bit processors has referred to having a 64bit processor (x86 is a given) and not an AMD64 architecture processor. Linus Torvalds addressed this specific issue for Linux and came down on the side of "x86-64": http://kerneltrap.org/node/2466 Look up AMD64 on Wikipedia and it redirects you to the X86-64 page. Information website setup by AMD and partners about the AMD64 architecture: http://www.x86-64.org/about.html In the AMD website they refer to "x86-64 Assembly": http://www.x86-64.org/documentation/assembly.html Microsoft seem to draw a distinction between x64 (which would also be acceptable) and Itanium based systems. Very rarely do they refer to AMD64: * http://www.microsoft.com/servers/64bit/compare.mspx * http://www.microsoft.com/servers/64bit/x64/overview.mspx * http://www.microsoft.com/servers/64bit/overview.mspx Using a vendor specific name automatically begs the question as to whether the installer works on processors from other vendors, as we saw in the specific enquiry from the user that triggered this debate. Referring to the AMD 64 build as x86-64, with a footnote explaining which architectures this specifically means is unlikely to confuse people. It is *definitely* better than just saying AMD64. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Thu Jan 14 02:50:55 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 19:50:55 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? Message-ID: <19278.30847.649228.115514@montanaro.dyndns.org> Just out of curiosity, is the static build stuff (use the old Modules/Setup file to build modules) exercised at all any more? Skip From benjamin at python.org Thu Jan 14 03:22:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 13 Jan 2010 20:22:03 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? In-Reply-To: <19278.30847.649228.115514@montanaro.dyndns.org> References: <19278.30847.649228.115514@montanaro.dyndns.org> Message-ID: <1afaf6161001131822r151bd202x35653ba44ee0235d@mail.gmail.com> 2010/1/13 : > > Just out of curiosity, is the static build stuff (use the old Modules/Setup > file to build modules) exercised at all any more? Exercised as in used in the build process? Yes. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Jan 14 10:23:47 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 09:23:47 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! Message-ID: <954450.26617.qm@web25804.mail.ukl.yahoo.com> In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: http://www.python.org/dev/peps/pep-0391/ In November 2009 I posted to this list that the PEP was ready for review. I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. Thanks & regards, Vinay Sajip From ziade.tarek at gmail.com Thu Jan 14 10:30:15 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Jan 2010 10:30:15 +0100 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <94bdd2611001140130u6154e893jfd37db32a25be66a@mail.gmail.com> On Thu, Jan 14, 2010 at 10:23 AM, Vinay Sajip wrote: [..] > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would > be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check > in the changes unless there are objections to doing so. All those who think that logging is currently > hard to configure will benefit from these changes, so here's the opportunity to pipe up and > improve things. FWIW, I am +1. Thanks for your work Tarek From hodgestar+pythondev at gmail.com Thu Jan 14 10:39:41 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 14 Jan 2010 11:39:41 +0200 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: On Thu, Jan 14, 2010 at 11:23 AM, Vinay Sajip wrote: > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I think the dict configuration scheme will be a great addition to the logging module. :) Schiavo Simon From p.f.moore at gmail.com Thu Jan 14 12:22:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jan 2010 11:22:09 +0000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> 2010/1/14 Vinay Sajip : > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I've no immediate need for the feature, but it would be good to have something like this, so I'm +1. Paul. From ncoghlan at gmail.com Thu Jan 14 12:46:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:46:19 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> Message-ID: <4B4F040B.8010607@gmail.com> Paul Moore wrote: > 2010/1/14 Vinay Sajip : >> So, can you please indicate your vote for or against incorporating PEP 391 into Python? > > I've no immediate need for the feature, but it would be good to have > something like this, so I'm +1. I'm in the same boat as Paul, but PEP 291 provides a nice forward compatible configuration scheme that will work with any application configuration approach that can produce an appropriate Python dictionary. So +1 from me too - I think the PEP has now taken this as far as theorising can go, and the only way to learn anything further is to put it into practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jan 14 12:57:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:57:55 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4B4F06C3.7030806@gmail.com> Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of > configuring logging using dictionaries: > > http://www.python.org/dev/peps/pep-0391/ Although one minor comment: you can probably remove the note about the "ext://" and "cfg://" prefixes being provisional at this stage. Those names look fine to me, so I don't see any point inviting a late-breaking bikeshed discussion about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu Jan 14 14:53:29 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 08:53:29 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> On Thu, Jan 14, 2010 at 4:23 AM, Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: > > ?http://www.python.org/dev/peps/pep-0391/ > > In November 2009 I posted to this list that the PEP was ready for review. > > I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. > > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. > > Thanks & regards, > > Vinay Sajip I'm generally +1 - but given I know that Django 1.2 is slated to implement something somewhat similar, I'm interested to hear how this proposal meshes with their plan(s). jesse From vinay_sajip at yahoo.co.uk Thu Jan 14 15:08:24 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 14:08:24 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> Message-ID: <92210.91656.qm@web25806.mail.ukl.yahoo.com> > From: Jesse Noller > I'm generally +1 - but given I know that Django 1.2 is slated to > implement something somewhat similar, I'm interested to hear how this > proposal meshes with their plan(s).. Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: http://groups.google.com/group/django-developers/msg/4ef81a2160257221 They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. Regards, Vinay Sajip From ssteinerx at gmail.com Thu Jan 14 15:54:27 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 14 Jan 2010 09:54:27 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <62E362ED-5DD7-4D42-B5E0-B263A137FD5C@gmail.com> On Jan 14, 2010, at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. That puts a huge +1 on there for me. If we can get this approved and have Django's logging improved *and* avoid a whole pile of duplicate effort in Django that's a huge win. S From jnoller at gmail.com Thu Jan 14 15:56:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 09:56:18 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <4222a8491001140656w68c7290agccfe7282cf96f2fb@mail.gmail.com> On Thu, Jan 14, 2010 at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. > > Regards, > > Vinay Sajip > Cool, +1 then :) From mal at egenix.com Thu Jan 14 20:19:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 20:19:12 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <4B4F6E30.50400@egenix.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users won't have PYTHONPATH or any other Python environment variables setup. Really, if you know that you are going to run Python 3 instead of Python 2 or vice-versa it's easy enough to run . py3env.sh or . py2env.sh in order to setup your shell environment, much like you typically do when using virtual Python installations or work on different projects that require different setups. If you just want to separate Python 2 and 3 files, use the user site-packages dir which includes the Python version. More experienced users could also write their own environment switching sitecustomize.py or usercustomize.py. And then there's the hackery option for experts that love dirty tricks: add a .pth file which includes an "import mysetup" line to fix-up you path and other settings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jan 14 22:02:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Jan 2010 07:02:28 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <4B4F8664.4080107@gmail.com> M.-A. Lemburg wrote: > Nick Coghlan wrote: >> Guido van Rossum wrote: >>> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >>> just removing the antiquated use of environment variables altogether >>> from Python 3 and avoid the issue completely. >>> >>> -1. They have their use, but more in controlled situations. If you >>> have "global" env vars that you only want to use with Python 2.x, >>> write a wrapper for Python 3 that invokes it with -E. >> Perhaps a case can be made for Python 3 to assume -E by default, with a >> -e option to enable reading of the environment variables? >> >> That way naive users could run Python3 without tripping over existing >> Python2 environment variables, while other tools could readily set up a >> different environment before launching Python3. > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. I was actually thinking of the situation where the OS had a preinstalled Python 2 with a default PYTHONPATH setting and the user was playing with Python 3. However, I agree that that is a fairly unlikely scenario (since preinstalled Pythons tend not to rely on the environment variables and a user sophisticated enough to be playing with a new Python interpreter should be able to cope with a few environment variable conflicts anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Thu Jan 14 22:09:30 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 21:09:30 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F8664.4080107@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <4B4F8664.4080107@gmail.com> Message-ID: <4B4F880A.5010809@voidspace.org.uk> On 14/01/2010 21:02, Nick Coghlan wrote: > However, I agree that that is a fairly unlikely scenario (since > preinstalled Pythons tend not to rely on the e Well, on the other hand I think that during the next few years it will be increasingly common for developers (and possibly users) to have Python 2 and Python 3 installed side-by-side. Many libraries and applications may never make the jump to Python 3 and Python users may be using 'legacy' Python 2 code for many years to come. It will also become increasingly common for developers to be using Python 3 *primarily* and for Python 3 only libraries and applications to emerge. Whilst there are workarounds we *are* in a situation that Python 2 and Python 3 share environment variables for the location of libraries and executing code on startup, whilst at the same time they are largely incompatible and need separate library paths and startup code. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ralf at brainbot.com Thu Jan 14 22:25:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Thu, 14 Jan 2010 22:25:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> (M.'s message of "Thu, 14 Jan 2010 20:19:12 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <871vhswf90.fsf@brainbot.com> "M.-A. Lemburg" writes: > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. > > Really, if you know that you are going to run Python 3 instead of > Python 2 or vice-versa it's easy enough to run You don't even know that you're going to run python. I have 40 python scripts in my /usr/bin directory. > > . py3env.sh > or > . py2env.sh > > in order to setup your shell environment, much like you typically > do when using virtual Python installations or work on different > projects that require different setups. No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > > If you just want to separate Python 2 and 3 files, use the > user site-packages dir which includes the Python version. Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to install those programs in a user's home directory. There are still people running python <2.6. From mal at egenix.com Thu Jan 14 22:51:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 22:51:07 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <871vhswf90.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <871vhswf90.fsf@brainbot.com> Message-ID: <4B4F91CB.2060106@egenix.com> Ralf Schmitt wrote: > "M.-A. Lemburg" writes: > >> >> Naive users won't have PYTHONPATH or any other Python environment >> variables setup. >> >> Really, if you know that you are going to run Python 3 instead of >> Python 2 or vice-versa it's easy enough to run > > You don't even know that you're going to run python. I have 40 python > scripts in my /usr/bin directory. > >> >> . py3env.sh >> or >> . py2env.sh >> >> in order to setup your shell environment, much like you typically >> do when using virtual Python installations or work on different >> projects that require different setups. > > No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > >> >> If you just want to separate Python 2 and 3 files, use the >> user site-packages dir which includes the Python version. > > Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to > install those programs in a user's home directory. There are still > people running python <2.6. Note that you only need to have a different PYTHONPATH setups for Python 2.x/3.x if you plan to run code that: * is not installed in site-packages (most OS shipped code will be found in the resp. system site-packages/ dir) * is not installed in a user site-packages directory (user installed code will typically go there (*)) * uses modules/packages that come in two different versions (one for Python 2.x and one for 3.x) and use the same name for both versions * needs to work in both Python 2.x and 3.x (*) The method for installing code in user site-packages dir is running: python setup.py install --user instead of the standard python setup.py install which install to the system-wide site-packages. BTW: I guess the bzr and mercurial wikis will need to be updated accordingly - at least for users of Python >=2.6. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Jan 14 22:55:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jan 2010 22:55:04 +0100 Subject: [Python-Dev] [ANN] Python 2.5.5 Release Candidate 1. Message-ID: <4B4F92B8.30806@v.loewis.de> On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.5. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.6 (which is 2.6.4 at this point). This releases fixes issues with the logging and tarfile modules, and with thread-local variables. See the detailed release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed. For more information on Python 2.5.5, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.5.5 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin at v.loewis.de Python Release Manager (on behalf of the entire python-dev team) From status at bugs.python.org Fri Jan 15 18:07:26 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Jan 2010 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100115170726.9963A7865F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/08/10 - 01/15/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2536 open (+32) / 16993 closed (+16) / 19529 total (+48) Open issues with patches: 1024 Average duration of open issues: 710 days. Median duration of open issues: 469 days. Open Issues Breakdown open 2501 (+32) pending 34 ( +0) Issues Created Or Reopened (53) _______________________________ PYTHON3PATH environment variable to supersede PYTHONPATH for mul 01/13/10 CLOSED http://bugs.python.org/issue2375 reopened r.david.murray Mark the compiler package as deprecated 01/13/10 http://bugs.python.org/issue6837 reopened ezio.melotti test_distutils failure 01/15/10 CLOSED http://bugs.python.org/issue6961 reopened flox buildbot test_urllib: unsetting missing 'env' variable 01/08/10 CLOSED http://bugs.python.org/issue7026 reopened flox Two float('nan') are not equal 01/08/10 CLOSED http://bugs.python.org/issue7660 created jmfauth compiling ctypes fails with non-ascii path 01/08/10 http://bugs.python.org/issue7661 created pitrou patch time.utcoffset() 01/09/10 http://bugs.python.org/issue7662 created techtonik UCS4 build incorrectly translates cases for non-BMP code points 01/10/10 CLOSED http://bugs.python.org/issue7663 created exarkun python logger does not handle IOError Exception 01/10/10 CLOSED http://bugs.python.org/issue7664 created yateenjoshi test_urllib2 and test_ntpath fail if path contains "\" 01/10/10 http://bugs.python.org/issue7665 created flox test_lib2to3 fails if path contains space 01/10/10 CLOSED http://bugs.python.org/issue7666 created flox test_doctest fails with non-ascii path 01/10/10 http://bugs.python.org/issue7667 created flox buildbot test_httpservers fails with non-ascii path 01/10/10 http://bugs.python.org/issue7668 created flox buildbot test_unicode_file fails with non-ascii path 01/10/10 CLOSED http://bugs.python.org/issue7669 created flox _sqlite3: Block *all* operations on a closed Connection object 01/10/10 http://bugs.python.org/issue7670 created haypo patch test_popen fails if path contains special char like ";" 01/11/10 http://bugs.python.org/issue7671 reopened flox patch _ssl module causes segfault 01/10/10 http://bugs.python.org/issue7672 created ssoria patch audioop: check that length is a multiple of the size 01/11/10 http://bugs.python.org/issue7673 created haypo patch select.select() corner cases: duplicate fds, out-of-range fds 01/11/10 http://bugs.python.org/issue7674 created cdleary help() doesn't accept unicode literals in built in docstrings 01/11/10 http://bugs.python.org/issue7675 created psd patch IDLE shell shouldn't use TABs 01/11/10 http://bugs.python.org/issue7676 created cben distutils, better error message for setup.py upload -sign withou 01/11/10 http://bugs.python.org/issue7677 created illume subprocess.Popen pipeline example code in the documentation is l 01/11/10 http://bugs.python.org/issue7678 created steven.k.wong Warning building 2.7 on OS X 10.6 libintl.h "Present But Cannot 01/12/10 http://bugs.python.org/issue7679 created ssteinerX pythonw crash while attempting to start() a thread object 01/12/10 http://bugs.python.org/issue7680 created dontbugme Incorrect division in [wave.py] 01/12/10 CLOSED http://bugs.python.org/issue7681 created alfps patch, needs review Optimisation of if with constant expression 01/12/10 http://bugs.python.org/issue7682 created sdefresne patch Wrong link in HTML documentation 01/12/10 CLOSED http://bugs.python.org/issue7683 created francescor decimal.py: infinity coefficients in tuples 01/12/10 http://bugs.python.org/issue7684 created skrah minor typo in re docs 01/12/10 CLOSED http://bugs.python.org/issue7685 created mikejs patch redundant open modes 'rbb', 'wbb', 'abb' no longer work on Windo 01/13/10 http://bugs.python.org/issue7686 created ivank Bluetooth support untested 01/13/10 http://bugs.python.org/issue7687 created pitrou TypeError: __name__ must be set to a string object 01/13/10 http://bugs.python.org/issue7688 created frankmillman Pickling of classes with a metaclass and copy_reg 01/13/10 http://bugs.python.org/issue7689 created craigcitro patch Wrong PEP number in importlib module manual page 01/13/10 CLOSED http://bugs.python.org/issue7690 created francescor test_bsddb3 files are not always removed when test fails 01/13/10 CLOSED http://bugs.python.org/issue7691 created flox buildbot Pointless comparision of unsigned integer with zero 01/13/10 CLOSED http://bugs.python.org/issue7692 created Bugger tarfile.extractall can't have unicode extraction path 01/13/10 http://bugs.python.org/issue7693 created pbienst DeprecationWarning from distuils.commands.build_ext needs stackl 01/13/10 http://bugs.python.org/issue7694 created tseaver patch missing termios constants 01/13/10 http://bugs.python.org/issue7695 created conorh Improve Memoryview/Buffer documentation 01/13/10 http://bugs.python.org/issue7696 created flox os.getcwd() should raise UnicodeDecodeError for arbitrary bytes 01/13/10 CLOSED http://bugs.python.org/issue7697 created flox pystack macro in Misc/gdbinit incorrectly uses PyEval_EvalFrame 01/14/10 CLOSED http://bugs.python.org/issue7698 created exarkun patch strptime, strftime documentation 01/14/10 http://bugs.python.org/issue7699 created mikejs patch "-3" flag does not work anymore 01/14/10 CLOSED http://bugs.python.org/issue7700 created flox patch fix output string length for binascii.b2a_uu() 01/14/10 CLOSED http://bugs.python.org/issue7701 created haypo patch Wrong order of parameters of _get_socket in SMTP class in smtpli 01/14/10 http://bugs.python.org/issue7702 created alito Replace buffer()-->memoryview() in Lib/ctypes/test/ 01/14/10 http://bugs.python.org/issue7703 created flox patch Math calculation problem (1.6-1.0)>0.6, python said TRUE 01/14/10 CLOSED http://bugs.python.org/issue7704 created DhaReaL libpython2.6.so is not linked correctly on FreeBSD when threads 01/15/10 http://bugs.python.org/issue7705 created aballier patch, needs review Missing #include guards 01/15/10 http://bugs.python.org/issue7706 created akrpic77 patch multiprocess.Queue operations during import can lead to deadlock 01/15/10 http://bugs.python.org/issue7707 created kripken Conversion of longs to bytes and vice-versa. 01/11/10 http://bugs.python.org/issue1023290 reopened mark.dickinson patch Issues Now Closed (67) ______________________ segfault in curses when calling redrawwin() before refresh() 825 days http://bugs.python.org/issue1266 mark.dickinson "RuntimeError: dictionary changed size during iteration" in Tkin 751 days http://bugs.python.org/issue1658 flox patch Exception exceptions.AttributeError '_shutdown' in 0.6, python said TRUE 0 days http://bugs.python.org/issue7704 tim_one Support for sending multipart form data 2452 days http://bugs.python.org/issue727898 r.david.murray email.MIME*.as_string removes duplicate spaces 1440 days http://bugs.python.org/issue1422094 r.david.murray easy test_parsedate_acceptable_to_time_functions+DST == :-( 1394 days http://bugs.python.org/issue1454285 r.david.murray email module does not complay with RFC 2046: CRLF issue 1196 days http://bugs.python.org/issue1571841 r.david.murray email.FeedParser.BufferedSubFile improperly handles "\r\n" 969 days http://bugs.python.org/issue1721862 r.david.murray patch, easy Top Issues Most Discussed (10) ______________________________ 20 dtoa.c: oversize b in quorem 11 days open http://bugs.python.org/issue7632 18 _ssl module causes segfault 5 days open http://bugs.python.org/issue7672 12 Speed up cPickle's pickling generally 287 days open http://bugs.python.org/issue5683 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 7 days open http://bugs.python.org/issue7658 8 Fatal error on thread creation in low memory condition 27 days open http://bugs.python.org/issue7544 8 Direct calls to PyObject_Del/PyObject_DEL are broken for --with 558 days open http://bugs.python.org/issue3299 7 "-3" flag does not work anymore 1 days closed http://bugs.python.org/issue7700 7 tarfile.extractall can't have unicode extraction path 2 days open http://bugs.python.org/issue7693 7 test_urllib: unsetting missing 'env' variable 0 days closed http://bugs.python.org/issue7026 7 os.path.abspath with unicode argument should call os.getcwdu 542 days open http://bugs.python.org/issue3426 From g.brandl at gmx.net Sat Jan 16 21:05:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 16 Jan 2010 21:05:46 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> Message-ID: Am 13.01.2010 21:27, schrieb Lennart Regebro: > On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: >> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >>> What do you need to do in the PYTHONSTARTUP file? >>> Ten years of Python programming, and I didn't even know it existed. :-) >> >> See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. > > Cheese and fries! :-) > > Anyway, I don't see how this could pose any problems, because any user > advanced enough to do these things will be advanced enough to > understand what the problem is and fix it so it works under Python 3 > too. I'd propose adding a bit of text to the environment variables documentation, and especially about the needs of PYTHONSTARTUP files. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From asmodai at in-nomine.org Sat Jan 16 21:50:56 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 16 Jan 2010 21:50:56 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <874ompg225.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> Message-ID: <20100116205056.GL99670@nexus.in-nomine.org> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >hehe. tab completion: With bpython and ipython available, why would you even want to stick to the 'plain old' interactive interpreter? (Sorry to derail the discussion, but maybe there's more people that have not heard of either or both.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B For ever, brother, hail and farewell... From solipsis at pitrou.net Sat Jan 16 21:57:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Jan 2010 20:57:13 +0000 (UTC) Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: Jeroen Ruigrok van der Werven in-nomine.org> writes: > > -On [20100113 22:13], Ralf Schmitt (ralf brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick to the > 'plain old' interactive interpreter? Why wouldn't we? There are probably many more people using the standard Python prompt than ipython. From ben+python at benfinney.id.au Sat Jan 16 23:41:03 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Jan 2010 09:41:03 +1100 Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: <87y6jxk70g.fsf@benfinney.id.au> Jeroen Ruigrok van der Werven writes: > -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick > to the 'plain old' interactive interpreter? Because those optional extras are not always available, whereas the standard interactive interpreter is always available with CPython distributions. -- \ ?I fly Air Bizarre. You buy a combination one-way round-trip | `\ ticket. Leave any Monday, and they bring you back the previous | _o__) Friday. That way you still have the weekend.? ?Steven Wright | Ben Finney From ncoghlan at gmail.com Sun Jan 17 01:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:22:10 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87y6jxk70g.fsf@benfinney.id.au> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> <87y6jxk70g.fsf@benfinney.id.au> Message-ID: <4B525832.8090904@gmail.com> Ben Finney wrote: > Jeroen Ruigrok van der Werven writes: > >> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >>> hehe. tab completion: >> With bpython and ipython available, why would you even want to stick >> to the 'plain old' interactive interpreter? > > Because those optional extras are not always available, whereas the > standard interactive interpreter is always available with CPython > distributions. I've never even contemplated the idea of installing 3rd party apps for the 4 current development builds (2.x trunk, 2.x maintenance, 3.x trunk, 3.x maintenance) on my home development machine. And of course work suffers from the problem of not being allowed to install arbitrary apps I downloaded from the internet without getting the licensing vetted for commercial acceptability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 01:34:08 2010 From: nad at acm.org (Ned Deily) Date: Sat, 16 Jan 2010 16:34:08 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? Message-ID: I've recently seen a couple of references to 3.1.2 go by in checkins which made me wonder whether dates have been proposed yet for updates to either 3.1 or 2.6. I don't recall seeing any and I didn't see any references in the PEPs. Some advance warning would be nice. I assume that some critical problem could trigger planning for an update on short notice; is there a time-limit trigger as well? -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Jan 17 01:55:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:55:38 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52600A.7060201@gmail.com> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Usually there is one more regular maintenance release of the existing maintenance branches within a few months of the creation of the next version (releases before then are at the discretion of the release manager, and security releases continue for much longer). So take the planned 2.7 and 3.2 release dates and add a couple of months to each one to get the likely release dates for the 2.6.x and 3.1.x maintenance releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sun Jan 17 10:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 17 Jan 2010 10:21:49 +0100 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52D6AD.6090302@v.loewis.de> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Barry was once proposing time-based releases; this idea didn't really catch on. It's the release manager who decides when the next bug fix release will be made, often in response to developers asking for one. Regards, Martin From solipsis at pitrou.net Sun Jan 17 14:00:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 13:00:04 +0000 (UTC) Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: Message-ID: Ned Deily acm.org> writes: > > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. There are a couple of release blockers right now. Once they are fixed or deferred, I think it would be nice to have a 3.1.2. Why do you need "some advance warning" though? From ncoghlan at gmail.com Sun Jan 17 14:40:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 23:40:42 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B53135A.7060104@gmail.com> Antoine Pitrou wrote: > Ned Deily acm.org> writes: >> I've recently seen a couple of references to 3.1.2 go by in >> checkins which made me wonder whether dates have been proposed yet >> for updates to either 3.1 or 2.6. I don't recall seeing any and I >> didn't see any references in the PEPs. Some advance warning would >> be nice. > > There are a couple of release blockers right now. Once they are fixed > or deferred, I think it would be nice to have a 3.1.2. Why do you > need "some advance warning" though? Advance warning does allow interested users that would consider upgrading to schedule time for testing before the maintenance release comes out. This is particularly useful in helping to make a 1-week RC period effective in picking up issues that might otherwise lead to a brown paper bag release to fix major issues that slipped through our own automated test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 19:01:37 2010 From: nad at acm.org (Ned Deily) Date: Sun, 17 Jan 2010 10:01:37 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: <4B53135A.7060104@gmail.com> Message-ID: In article <4B53135A.7060104 at gmail.com>, Nick Coghlan wrote: > Antoine Pitrou wrote: > > Ned Deily acm.org> writes: > >> I've recently seen a couple of references to 3.1.2 go by in > >> checkins which made me wonder whether dates have been proposed yet > >> for updates to either 3.1 or 2.6. I don't recall seeing any and I > >> didn't see any references in the PEPs. Some advance warning would > >> be nice. > > There are a couple of release blockers right now. Once they are fixed > > or deferred, I think it would be nice to have a 3.1.2. Why do you > > need "some advance warning" though? > > Advance warning does allow interested users that would consider > upgrading to schedule time for testing before the maintenance release > comes out. This is particularly useful in helping to make a 1-week RC > period effective in picking up issues that might otherwise lead to a > brown paper bag release to fix major issues that slipped through our own > automated test coverage. That. and resource contention: there are always potential fixes in the pipeline that could or should be bumped in priority if one knows there is a code cutoff approaching. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Sun Jan 17 20:51:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 20:51:58 +0100 Subject: [Python-Dev] Enhancing the shutil module Message-ID: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Hello, For 2.7/3.2, I am in the process of removing modules in Distutils that can be replaced by calls to existing functions in stdlib. For instance, "dir_util" and "file_util" (old modules from the Python 1.x era) are going away in favor of calls to shutil (and os), so the Distutils package gets lighter. Another module I would like to move away from Distutils is "archive_util". It contains helpers to build archives, whether they are zip or tar files. I propose to move those useful functions into shutil, as this seems the most logical place. I also propose to maintain this "shutil" module for now on (no one is declared as a maintainer in maintainers.rst) since Distutils will become a heavy user of its functions. Any objections/opinions ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 20:53:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 19:53:03 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536A9F.5050203@voidspace.org.uk> On 17/01/2010 19:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > > Regards, > Tarek > > I think it's a great idea. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Sun Jan 17 20:55:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2010 11:55:47 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > If it's archive-agnostic then shutil is probably the best place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Great! > Any objections/opinions ? > No objections from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jan 17 21:04:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 20:04:41 +0000 (UTC) Subject: [Python-Dev] Enhancing the shutil module References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: Tarek Ziad? gmail.com> writes: > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. Are these functions portable? Do they rely on external programs? > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. It's ok for me. Regards Antoine. From ziade.tarek at gmail.com Sun Jan 17 21:09:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:09:18 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: > > > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >> >> Hello, >> >> For 2.7/3.2, I am in the process of removing modules in Distutils that >> can be replaced by calls to existing functions in stdlib. For >> instance, "dir_util" and "file_util" (old modules from the Python 1.x >> era) are going away in favor of calls to shutil (and os), so the >> Distutils package gets lighter. >> >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. > > If it's archive-agnostic then shutil is probably the best place. In more details: It allows the creation of gzip, bzip2, tar and zip files through a single API. There's a registry of supported formats and the API is driven by a format identifier. To do the work it uses stdlib's compression modules. Although it tries the "zip" system command as a fallback if the "zipfile" module is not present. (notice that I've removed the support of "compress" (.Z) some time ago) Regards Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 21:15:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 20:15:00 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536FC4.9090304@voidspace.org.uk> On 17/01/2010 20:04, Antoine Pitrou wrote: > Tarek Ziad? gmail.com> writes: > >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. >> > Are these functions portable? Do they rely on external programs? > > I believe that part of the work that Tarek has been doing has been to make these distutils commands use the Python standard library and not depend on external programs. In which case they seem like *excellent* additions to the shutil module. Of course Tarek can speak for himself... Michael >> I also propose to maintain this "shutil" module for now on (no one is >> declared as a maintainer in maintainers.rst) since Distutils will >> become a heavy user of its functions. >> > It's ok for me. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Sun Jan 17 21:43:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:43:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B536FC4.9090304@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> Message-ID: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> On Sun, Jan 17, 2010 at 9:15 PM, Michael Foord wrote: [..] >> Are these functions portable? Do they rely on external programs? >> >> > > I believe that part of the work that Tarek has been doing has been to make > these distutils commands use the Python standard library and not depend on > external programs. In which case they seem like *excellent* additions to the > shutil module. yes, in the past the "tar" files where created using the "tar" command but this has been changed. For a while now, they are portable and they use stdlib code only. A recent addition is to be able to define user/group permissions in the tar files, thanks to Lars' work in the tarfile module. There's one remaining external call for "zip" done if the zip module is not found, but I am happy to remove it and throw an exception if it's not found, and keep the external "zip" call on Distutils side, so shutil stays 100% stdlib-powered. > Of course Tarek can speak for himself... Thanks for explaining it ! :) Regards Tarek -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Sun Jan 17 22:50:52 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sun, 17 Jan 2010 13:50:52 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> Message-ID: <4B53863C.5060304@activestate.com> On 1/17/2010 12:09 PM, Tarek Ziad? wrote: > On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: >> > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >>> >> Another module I would like to move away from Distutils is >>> >> "archive_util". It contains helpers to build archives, whether they >>> >> are zip or tar files. I propose to move those useful functions into >>> >> shutil, as this seems the most logical place. >> > If it's archive-agnostic then shutil is probably the best place. > In more details: > It allows the creation of gzip, bzip2, tar and zip files through a single API. > There's a registry of supported formats and the API is driven by a > format identifier. Will it also allow decompression of the said archive types? Distribute has some utility code to handle zip/tar archives. So does PyPM. This is because the `tarfile` and `zipfile` modules do not "just work" due to several issues. See http://gist.github.com/279606 Take note of the following in the above code: 1) _ensure_read_write_access 2) *File.is_valid 3) ZippedFile.extract ... issue 6510 4) ZippedFile.extract ... issue 6609 5) TarredFile.extract ... issue 6584 6) The way unpack() detects the unpacked directory. -srid From ziade.tarek at gmail.com Sun Jan 17 23:09:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 23:09:29 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B53863C.5060304@activestate.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> <4B53863C.5060304@activestate.com> Message-ID: <94bdd2611001171409j4ec1e0bfj47e59894fdec0d8a@mail.gmail.com> On Sun, Jan 17, 2010 at 10:50 PM, Sridhar Ratnakumar wrote: [..] > Will it also allow decompression of the said archive types? No but it would make sense having this one as well. Distribute/Setuptools' "unpack_archive" (used by easy_install) was implemented using the same principle as Distutils' "make_archive". I can add it as well while I am at it : it has been successfully used for years by easy_install. > Distribute has > some utility code to handle zip/tar archives. So does PyPM. This is because > the `tarfile` and `zipfile` modules do not "just work" due to several > issues. > > See http://gist.github.com/279606 > > Take note of the following in the above code: > > ?1) _ensure_read_write_access > ?2) *File.is_valid > ?3) ZippedFile.extract ... issue 6510 > ?4) ZippedFile.extract ... issue 6609 > ?5) TarredFile.extract ... issue 6584 Looks like some of these are already fixed (I just looked quickly in the bug tracker though) If its not already done and pending, it would be great if you could refactor your fixes into patches for the remaining issues for each one of those modules Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Sun Jan 17 23:56:37 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 17 Jan 2010 17:56:37 -0500 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: <4B52D6AD.6090302@v.loewis.de> References: <4B52D6AD.6090302@v.loewis.de> Message-ID: On Jan 17, 2010, at 4:21 AM, Martin v. L?wis wrote: > Barry was once proposing time-based releases; this idea didn't really > catch on. I'm still a proponent of this, but it doesn't seem like there's enough support for it. > It's the release manager who decides when the next bug fix release will > be made, often in response to developers asking for one. I'm happy to make a 2.6.5 release whenever it makes sense. -Barry From ncoghlan at gmail.com Mon Jan 18 13:40:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Jan 2010 22:40:01 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> Message-ID: <4B5456A1.2080109@gmail.com> Tarek Ziad? wrote: > There's one remaining external call for "zip" done if the zip module > is not found, but I am happy to remove it and throw an exception if > it's not found, and keep the external "zip" call on Distutils side, so > shutil stays 100% stdlib-powered. +1 for that approach. These changes all sound like nice additions to shutil, and hooray for every module that gets adopted by an active maintainer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From masklinn at masklinn.net Mon Jan 18 14:34:14 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 14:34:14 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5456A1.2080109@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> Message-ID: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: > > Tarek Ziad? wrote: >> There's one remaining external call for "zip" done if the zip module >> is not found, but I am happy to remove it and throw an exception if >> it's not found, and keep the external "zip" call on Distutils side, so >> shutil stays 100% stdlib-powered. > > +1 for that approach. These changes all sound like nice additions to > shutil, and hooray for every module that gets adopted by an active > maintainer :) Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. From doug.hellmann at gmail.com Mon Jan 18 14:46:05 2010 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 18 Jan 2010 08:46:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils >>> side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil > advertises itself as "a number of high-level operations on files and > collections of files." and from what I understood it was a bunch of > shell-type utility functions to easily copy, move or remove files > and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/ > objects in a new module separate from shutil, dealing specifically > with cross-archive APIs and linked from the current archive-specific > modules (essentially, just take the current archive_util, move it to > the toplevel of the stdlib and maybe rename it)? It would also make > the module much easier to find when searching through the module > listing, I think. +1 From fuzzyman at voidspace.org.uk Mon Jan 18 14:57:49 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 18 Jan 2010 13:57:49 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> Message-ID: <4B5468DD.5040200@voidspace.org.uk> On 18/01/2010 13:46, Doug Hellmann wrote: > > On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > >> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>> >>> Tarek Ziad? wrote: >>>> There's one remaining external call for "zip" done if the zip module >>>> is not found, but I am happy to remove it and throw an exception if >>>> it's not found, and keep the external "zip" call on Distutils side, so >>>> shutil stays 100% stdlib-powered. >>> >>> +1 for that approach. These changes all sound like nice additions to >>> shutil, and hooray for every module that gets adopted by an active >>> maintainer :) >> >> Isn't it a bit weird to include that to shutil though? shutil >> advertises itself as "a number of high-level operations on files and >> collections of files." Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> and from what I understood it was a bunch of shell-type utility functions like tar and zip? :-) >> to easily copy, move or remove files and directories (that's pretty >> much all there is in it at this time). >> I don't think the additions are out of place prima-facie. >> Wouldn't it make more sense to put those "archive utils" >> functions/objects in a new module separate from shutil, dealing >> specifically with cross-archive APIs and linked from the current >> archive-specific modules (essentially, just take the current >> archive_util, move it to the toplevel of the stdlib and maybe rename >> it)? It would also make the module much easier to find when searching >> through the module listing, I think. > > +1 > Proliferation of modules is itself a bad thing though. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Mon Jan 18 15:34:12 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 15:34:12 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <94bdd2611001180634sacd998fm6f67dc680e2e61c3@mail.gmail.com> On Mon, Jan 18, 2010 at 2:57 PM, Michael Foord wrote: [..] >>> Wouldn't it make more sense to put those "archive utils" >>> functions/objects in a new module separate from shutil, dealing specifically >>> with cross-archive APIs and linked from the current archive-specific modules >>> (essentially, just take the current archive_util, move it to the toplevel of >>> the stdlib and maybe rename it)? It would also make the module much easier >>> to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. I am with Michael here. I think having this function in shutil is the right place. For the find problem, I think docs.python.org is easy to search now, as long as the shutil module documentation has an good set of examples for this new API. We could even add references in the tarfile, zipfile modules documentation pointing to these examples. Regards Tarek -- Tarek Ziad? | http://ziade.org From masklinn at masklinn.net Mon Jan 18 15:56:05 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 15:56:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> On 18 Jan 2010, at 14:57 , Michael Foord wrote: > > On 18/01/2010 13:46, Doug Hellmann wrote: >> >> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: >> >>> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>>> >>>> Tarek Ziad? wrote: >>>>> There's one remaining external call for "zip" done if the zip module >>>>> is not found, but I am happy to remove it and throw an exception if >>>>> it's not found, and keep the external "zip" call on Distutils side, so >>>>> shutil stays 100% stdlib-powered. >>>> >>>> +1 for that approach. These changes all sound like nice additions to >>>> shutil, and hooray for every module that gets adopted by an active >>>> maintainer :) >>> >>> Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." > > Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? > Well no those are high-level operations on a very restricted set of file types (archives). >>> and from what I understood it was a bunch of shell-type utility functions > > like tar and zip? :-) > Tar and zip have a module each at this time, so they're considered pretty big. I don't think anybody would consider "mv" big enough to give it a module on its own. >>> Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. From phd at phd.pp.ru Mon Jan 18 16:03:38 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 18 Jan 2010 18:03:38 +0300 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118150337.GA19391@phd.pp.ru> On Mon, Jan 18, 2010 at 02:34:14PM +0100, Masklinn wrote: > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. +1 Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ziade.tarek at gmail.com Mon Jan 18 16:24:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 16:24:37 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> Message-ID: <94bdd2611001180724u7f48e620ub32e13ef96c873b9@mail.gmail.com> On Mon, Jan 18, 2010 at 3:56 PM, Masklinn wrote: [..] >> Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> > Well no those are high-level operations on a very restricted set of file types (archives) not really: make_archive/unpack_archive, are also dealing with files and directories. [..] >> Proliferation of modules is itself a bad thing though. > Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. > > Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. I am not sure why this would happen. For instance, shutil is already on the top of the os module since a few major versions IIRC, because it reads and writes files and directories. But it was not moved into the os package (or vice-versa) The shutil module uses APIs to read and write files. So if it works with archives, it's just a specific read/write API that is used, but that doesn't mean tarfile and zipfile might be reunited with shutil imho. If the shutil module is restricted to high-level files and directories manipulation, working with archives is just a target like another I think. But at the end I am 0- to create a new module, because what really matters to me is to take it out of Distutils :) Regards, Tarek -- Tarek Ziad? | http://ziade.org From listsin at integrateddevcorp.com Mon Jan 18 16:56:05 2010 From: listsin at integrateddevcorp.com (Steve Steiner (listsin)) Date: Mon, 18 Jan 2010 10:56:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. As much of a pain as it is to get new modules accepted, I agree that mixing archiving functions into shutil is not the right way to do it and that a separate archive_util module would make much more sense and would give a logical place to put any extensions to archive handling. S From rdmurray at bitdance.com Mon Jan 18 20:28:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 18 Jan 2010 14:28:47 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: > As much of a pain as it is to get new modules accepted, I agree that > mixing archiving functions into shutil is not the right way to do it > and that a separate archive_util module would make much more sense and > would give a logical place to put any extensions to archive handling. Looking at the source code and API for both shutil and archive_util, I think that the archive_util methods fit into shutil. shutil currently wraps some standard library facilities with convenience functions for operations you might otherwise perform at the shell command line using OS facilities. As far as I can tell, archive_util does the same, and seems quite within the shutil mission of "high level file operations". So +1 from me for putting these in shutil. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From p.f.moore at gmail.com Mon Jan 18 20:48:43 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 18 Jan 2010 19:48:43 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> Message-ID: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> 2010/1/18 R. David Murray : > On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: >> As much of a pain as it is to get new modules accepted, I agree that >> mixing archiving functions into shutil is not the right way to do it >> and that a separate archive_util module would make much more sense and >> would give a logical place to put any extensions to archive handling. > > Looking at the source code and API for both shutil and archive_util, I > think that the archive_util methods fit into shutil. ?shutil currently > wraps some standard library facilities with convenience functions > for operations you might otherwise perform at the shell command line using > OS facilities. ?As far as I can tell, archive_util does the same, and > seems quite within the shutil mission of "high level file operations". > > So +1 from me for putting these in shutil. Conceptually, I'm happy with these going into shutil (and +1 on the rest of Tarek's proposal, too!) To my mind, shutil is a module for higher-level operations on files - the sort of things you'd do in shell commands, like move a batch of files around (mv), create a directory tree (mkdir -p). Tarring or zipping up a batch of files fits nicely into that space. Paul. From david.lyon at preisshare.net Tue Jan 19 02:53:43 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 18 Jan 2010 20:53:43 -0500 Subject: [Python-Dev] PyCon Keynote In-Reply-To: References: Message-ID: <0dfb370163cf3a284ca256f49662db74@preisshare.net> On Wed, 13 Jan 2010 10:51:14 -0800, Guido van Rossum wrote: > Please mail me topics you'd like to hear me talk about in my keynote > at PyCon this year. As a typical (but perphaps more vocal) python developer I would like to hear things that might aspire me and others to move to python 3. The way I see it is that python 2.x represents everyones comfort zone. It's safe and nobody got fired at work for using python 2.x I would like to hear how python 3 could be 'shaken' up with a slightly less conservative policy that has gone with the python 2.x series. The logic perhaps being that since only a minority use the 3.x series, it's functionality set is still more up in the air. imo it needs bigger batteries to give it more power than the 2.x series. This meaning that the stdlib could take an extra 5-10 packages not in the 2.x series. Just as one example, sphinx - a great documentation tool. I can easily name five or six others. I'd also like to hear more of your ideas on pypi and getting it to have as much who-ha as CPAN. You could do a lot to enlarge the developer group. Help us all get our priorities to be inline with your own wishes and expectations. If you ask us all to put in a big year and buy you a beer at the end.. I'm certain we all will. Every little bit of inspiration and direction counts for a lot... Best Regards David From ncoghlan at gmail.com Tue Jan 19 12:20:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Jan 2010 21:20:05 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> Message-ID: <4B559565.8090602@gmail.com> Paul Moore wrote: > 2010/1/18 R. David Murray : >> So +1 from me for putting these in shutil. > > Conceptually, I'm happy with these going into shutil (and +1 on the > rest of Tarek's proposal, too!) > > To my mind, shutil is a module for higher-level operations on files - > the sort of things you'd do in shell commands, like move a batch of > files around (mv), create a directory tree (mkdir -p). Tarring or > zipping up a batch of files fits nicely into that space. This is also reflected in the way at least Windows handles archives these days - it took them a couple of iterations to get it right (and resolve some of the performance impacts), but Explorer now does a decent job of integrating archives into the directory tree as "folders that happen to be compressed". Are archives as fundamental as directories and files? No. But in the context of shutil, the fact that their internal structure is largely about directories and files makes them more than just another arbitrary file type. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 19 14:16:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 08:16:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad Message-ID: <20100119081639.5d431ed9@freewill> I've just updated the Launchpad mirrors for the 4 active Python branches, trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar branches on code.python.org but it's probably been 7 months or so since those were regularly updated. Now the Launchpad branches sync against the read-only Subversion branches at http://svn.python.org, so they should remain up-to-date (within the re-sync timeframe of about 4 hours). This means you can once again use Bazaar to get local branches of Python, and you can of course push your own branches to Launchpad. I believe you can even use the bzr-svn plugin to commit changes back to the Subversion master, though I have not yet tried this. To get a local branch, just do any of the following: % bzr branch lp:python (for trunk) % bzr branch lp:python/2.6 % bzr branch lp:python/py3k % bzr branch lp:python/3.1 (It's fairly easy to create new mirrors for other Subversion branches, e.g. Python 2.5; just drop me an email if you want them.) If you're going to create a lot of branches you probably want to put them in a shared repository. E.g. % bzr init-repo pythonbzr % cd pythonbzr % bzr branch lp:python/py3k Bazaar 2.0 or better is recommended. For me, it took about 5m to check the first branch out from Launchpad, and then about 30s or so for each subsequent branch. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From abpillai at gmail.com Tue Jan 19 14:37:18 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Tue, 19 Jan 2010 19:07:18 +0530 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <8548c5f31001190537l2bcab5cfkb6f461910e4abd8b@mail.gmail.com> On Mon, Jan 18, 2010 at 1:21 AM, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > +1 for this. Just make sure that you change the docstring of shutil which now reads as, " shutil - Utility functions for copying files and directory trees." According to this "definition", archives don't fit in there. But the functionality does fit right in, so just need to make sure that it is reflected in the __doc__ . > Regards, > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jan 19 16:50:42 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Jan 2010 15:50:42 +0000 (UTC) Subject: [Python-Dev] Mailing List archive corruption? Message-ID: Hi, When I look at the mailing list archive for python-dev, I see some odd stuff at the bottom of the page: http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 Anyone know what's happened? Regards, Vinay Sajip From barry at python.org Tue Jan 19 17:07:48 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:07:48 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: Message-ID: <20100119110748.69bc564a@freewill> On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: >When I look at the mailing list archive for python-dev, I see some odd stuff at >the bottom of the page: > >http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >Anyone know what's happened? WTF? I think the archives were recently regenerated, so there's probably a fubar there. CC'ing the postmasters. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From foom at fuhm.net Tue Jan 19 17:24:57 2010 From: foom at fuhm.net (James Y Knight) Date: Tue, 19 Jan 2010 11:24:57 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119110748.69bc564a@freewill> References: <20100119110748.69bc564a@freewill> Message-ID: On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > >> When I look at the mailing list archive for python-dev, I see some >> odd stuff at >> the bottom of the page: >> >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 >> >> Anyone know what's happened? > > WTF? I think the archives were recently regenerated, so there's > probably a > fubar there. CC'ing the postmasters. That happens if messages had unescaped "From" lines in the middle of them. No doubt, you've now broken every link anyone had ever made into the python-dev archives, because now all the article numbers are different. BTDT...unfortunately... Pipermail really is quite crappy, sigh. Anyhow, when I did that, I went back to a backup to get the original article numbers, and edited the mbox file escaping From lines or adding additional empty messages until the newly regenerated article numbers matched the originals. I'd highly recommend going through that painful process, since I suspect a *lot* of people have links to the python-dev archive. Hope you have a backup (or can find caches on google or archive.org or something). James From barry at python.org Tue Jan 19 17:44:21 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:44:21 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119114421.3b96fbd4@freewill> On Jan 19, 2010, at 11:24 AM, James Y Knight wrote: >No doubt, you've now broken every link anyone had ever made into the >python-dev archives, because now all the article numbers are >different. BTDT...unfortunately... Pipermail really is quite crappy, >sigh. I've been trying for 10+ years to get folks interested in helping me fix this (and a few other warts) in Pipermail, to not much success. ;/ >Anyhow, when I did that, I went back to a backup to get the original >article numbers, and edited the mbox file escaping From lines or >adding additional empty messages until the newly regenerated article >numbers matched the originals. I'd highly recommend going through that >painful process, since I suspect a *lot* of people have links to the >python-dev archive. Hope you have a backup (or can find caches on >google or archive.org or something). bin/cleanarch uses a set of heuristics to find unescaped From lines and fix them. It's generally pretty good, but it certain can change message numbers (and sadly, their urls). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rdmurray at bitdance.com Tue Jan 19 18:48:41 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 19 Jan 2010 12:48:41 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119174841.9BC3B16A53@kimball.webabinitio.net> On Tue, 19 Jan 2010 11:24:57 -0500, James Y Knight wrote: > > On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > > > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > > > >> When I look at the mailing list archive for python-dev, I see some > >> odd stuff at the bottom of the page: > >> > >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >> > >> Anyone know what's happened? > > > > WTF? I think the archives were recently regenerated, so there's > > probably a fubar there. CC'ing the postmasters. > > That happens if messages had unescaped "From" lines in the middle of > them. > > No doubt, you've now broken every link anyone had ever made into the > python-dev archives, because now all the article numbers are > different. BTDT...unfortunately... Pipermail really is quite crappy, > sigh. > > Anyhow, when I did that, I went back to a backup to get the original > article numbers, and edited the mbox file escaping From lines or > adding additional empty messages until the newly regenerated article > numbers matched the originals. I'd highly recommend going through that > painful process, since I suspect a *lot* of people have links to the > python-dev archive. Hope you have a backup (or can find caches on > google or archive.org or something). The Python issue tracker does, for one. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From ncoghlan at gmail.com Tue Jan 19 22:18:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 07:18:52 +1000 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119174841.9BC3B16A53@kimball.webabinitio.net> References: <20100119110748.69bc564a@freewill> <20100119174841.9BC3B16A53@kimball.webabinitio.net> Message-ID: <4B5621BC.4070608@gmail.com> R. David Murray wrote: > The Python issue tracker does, for one. And all the PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at pythontest.org Wed Jan 20 00:16:44 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 10:16:44 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119081639.5d431ed9@freewill> References: <20100119081639.5d431ed9@freewill> Message-ID: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Hi Barry, That looks very interesting... So does that mean we could update the stdlib for a given python version using this ? David > I've just updated the Launchpad mirrors for the 4 active Python branches, > trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar > branches > on code.python.org but it's probably been 7 months or so since those were > regularly updated. Now the Launchpad branches sync against the read-only > Subversion branches at http://svn.python.org, so they should remain > up-to-date > (within the re-sync timeframe of about 4 hours). > > This means you can once again use Bazaar to get local branches of Python, > and > you can of course push your own branches to Launchpad. I believe you can > even > use the bzr-svn plugin to commit changes back to the Subversion master, > though > I have not yet tried this. > > To get a local branch, just do any of the following: > > % bzr branch lp:python (for trunk) > % bzr branch lp:python/2.6 > % bzr branch lp:python/py3k > % bzr branch lp:python/3.1 > > (It's fairly easy to create new mirrors for other Subversion branches, > e.g. Python 2.5; just drop me an email if you want them.) > > If you're going to create a lot of branches you probably want to put them > in a > shared repository. E.g. > > % bzr init-repo pythonbzr > % cd pythonbzr > % bzr branch lp:python/py3k > > Bazaar 2.0 or better is recommended. For me, it took about 5m to check > the > first branch out from Launchpad, and then about 30s or so for each > subsequent > branch. > > Enjoy, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/david.lyon%40pythontest.org > From barry at python.org Wed Jan 20 00:54:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 18:54:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119185439.299660c7@freewill> On Jan 20, 2010, at 10:16 AM, David Lyon wrote: >Hi Barry, > >That looks very interesting... Hi David, >So does that mean we could update the stdlib for a given >python version using this ? In a sense, yes (if I understand your question correctly). You can use Bazaar to branch any of the 4 Python series and use all the modern DVCS goodness to develop your updates. You can share your branches with others via e.g. Launchpad and even request reviews (called "merge proposals") to get feedback from others. The one thing I am unsure about, mostly because I have not tried it, is whether your Bazaar branch can be used to commit directly back to the Python Subversion master branches. I /think/ the answer is yes, assuming of course that you have permission to do so, and that you have a modern version of Bazaar and the bzr-svn plugin. It might even be possible to commit your Bazaar branch to a local Subversion branch, and then commit the latter to get it pushed up to svn.python.org. At worst, you would use Bazaar's features to get your patch into a state you and your reviewers are happy with, then you would generate a diff for application to your copy of the svn.python.org branch. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 01:51:23 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 11:51:23 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119185439.299660c7@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> Message-ID: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> > On Jan 20, 2010, at 10:16 AM, Barry wrote: >> So does that mean we could update the stdlib for a given >> python version using this ? > > In a sense, yes (if I understand your question correctly). Yeah, it just needs an implementation. > The one thing I am unsure about, mostly because I have not tried it, is > whether your Bazaar branch can be used to commit directly back to the > Python Subversion master branches. I /think/ the answer is yes, > assuming of course that you have permission to do so... Well I'm too Senior and my stuff is too forward looking to qualify for that just yet. I'd be happy to see bzr and mercurial and git all made it together into the stdlib for python 3. That would give a superb updating mechanism for python that would propel python well beyond the dinosaur badlands of CPAN and other languages. I was actually reading from (http://en.wikipedia.org/wiki/Python_%28programming_language%29): "Rather than requiring all desired functionality to be built into the language's core, Python was designed to be highly extensible. .. .. This design of a small core language with a large standard library and an easily extensible interpreter was intended by Van Rossum from the very start because of his frustrations with ABC (which espoused the opposite mindset).[5]" To me, the source code control systems seem to be fully in tune with the original design of python. That is, to be able to easily pull external libraries in. I think what has changed is that the mechanisms now (the SCMs) are way more highly developed than before. Apart from that though, after reading the full wikipedia article I'm left with the distinct impression that things are still pretty much the same (in that python design philosophy is advanced), just that the landscape (of external C libraries) has changed. Now all the libraries are external (on the internet) and all externally managed. So with just a tiny amount of work, imho we could pull it all together to bring python 3 *back* to being that cool tool that it once was (not saying it isn't now). Were you offering me an experimental branch somewhere for python 3 SCM integration ? David From jnoller at gmail.com Wed Jan 20 02:09:15 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 20:09:15 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> On Tue, Jan 19, 2010 at 7:51 PM, David Lyon wrote: >> On Jan 20, 2010, at 10:16 AM, Barry wrote: > >>> So does that mean we could update the stdlib for a given >>> python version using this ? >> >> In a sense, yes (if I understand your question correctly). > > Yeah, it just needs an implementation. > >> The one thing I am unsure about, mostly because I have not tried it, is >> whether your Bazaar branch can be used to commit directly back to the >> Python Subversion master branches. ?I /think/ the answer is yes, >> assuming of course that you have permission to do so... > > Well I'm too Senior and my stuff is too forward looking to qualify > for that just yet. > > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. i sincerely doubt that a source control system will be included in the standard library in the future. Especially 3. A SCM is not a "package management system". Barry was talking about mirrors of the python code. It is true a "package manager" could be developed based on a SCM, however you need to implement this far away from the stdlib and get traction with it within the community long before inclusion would be considered. The decision to move python's source control from SVN to mercurial was controversial enough; including 3 or more scm libraries into core would be an intractable uphill mountain of bike sheds. > So with just a tiny amount of work, imho we could pull > it all together to bring python 3 *back* to being that > cool tool that it once was (not saying it isn't now). Python 3 is still modularized, still has a standard library, etc. If you're really interested in helping with the standard library, get on stdlib-sig, and get ready to write code and PEPs. > Were you offering me an experimental branch somewhere > for python 3 SCM integration ? Barry made bzr mirrors of the python svn tree. Not a python with bzr included. jesse From barry at python.org Wed Jan 20 04:32:30 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 22:32:30 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <20100119223230.4c4a7ed5@freewill> On Jan 19, 2010, at 08:09 PM, Jesse Noller wrote: >The decision to move python's source control from SVN to mercurial was >controversial enough; including 3 or more scm libraries into core >would be an intractable uphill mountain of bike sheds. I'd be surprised if any of the big 3 DVCS developers would actually /want/ their stuff in the stdlib. Being in the stdlib has its advantages and disadvantages. I think for rapidly developing technology, the latter can actually outweigh the former. (Besides, git in the stdlib doesn't make much sense :). >Barry made bzr mirrors of the python svn tree. Not a python with bzr included. Bingo. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 04:43:12 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 14:43:12 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> > On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: > Python 3 is still modularized, still has a standard library, etc. If > you're really interested in helping with the standard library, get on > stdlib-sig, and get ready to write code and PEPs. Thank you for your direction to move these items forward to PEPs and Code. > i sincerely doubt that a source control system will be included in the > standard library in the future. Especially 3. Yeah and who twenty years ago thought you would get a 1GB memory card for $3 when all we had was 10Meg hard disks and they were the full 8" platter. > A SCM is not a "package management system". Exactly. It almost makes the need for a "package management system" pretty much obsolete if you can update your code directly from the developers sources. That's what all these SCMs provide. Plus it's addictive. It's hard to go back to 'package' style technology once you have all your code on an SCM based feed. > Barry was talking about mirrors of the python code. It is true a > "package manager" could be developed based on a SCM, however you need > to implement this far away from the stdlib and get traction with it > within the community long before inclusion would be considered. I think I'll have better chances with PEPs. Being honest, if wonderful libraries like Sphinx and Mercurial and Git and BZR can't make it into the stdlib, then there is no hope for even newer code to get in there. Plus, promoting all sorts of new and fangled tools however good they may or may not be just confuses users and ends up being a waste of time imho. It isn't good management of volounteers time and effort. If you could imagine disaster relief coordinated this way, it would just be a disaster in itself. That's why it has taken some 5 years to get PEP-345 done. > The decision to move python's source control from SVN to mercurial was > controversial enough; including 3 or more scm libraries into core > would be an intractable uphill mountain of bike sheds. Not at all. It would be a very fair thing to do. Not to mention being great for users. > Barry made bzr mirrors of the python svn tree. Not a python with bzr > included. I can't resist asking for that again.. I heard it only in Monty speek. Did you just say ?: "Barry made a bizarre mirror of the python suvern tree. Not a python with a buzzer included." Anyway.. Maybe I do get what your talking about. Even if you do talk with a strange accent. :-) David From jnoller at gmail.com Wed Jan 20 05:10:07 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 23:10:07 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001192010v66b728dbxa7ad1ed1fc1df15f@mail.gmail.com> On Tue, Jan 19, 2010 at 10:43 PM, David Lyon wrote: [snip] > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. Did you ever stop to think that some package authors do not want their code in the standard library? That throwing random shiny things in there just makes it a junk drawer? Besides: Show us the PEP to include Sphinx, and it's dependencies in the standard lib, with Georg's signature at the bottom. The authors of modules have to want things to be in there, they have to be best of breed, tested or common-enough patterns to warrant a slot. We have things in the standard lib which still need more TLC and maintenance, or they need to be shunted into space. Adding random things, which may or may not help packaging, and installing things from random places in someone source repository won't make things better. > Plus, promoting all sorts of new and fangled tools however > good they may or may not be just confuses users and ends > up being a waste of time imho. It isn't good management > of volounteers time and effort. > > If you could imagine disaster relief coordinated this > way, it would just be a disaster in itself. > > That's why it has taken some 5 years to get PEP-345 done. I'm going to assume that you're trolling now, or intentionally misrepresenting facts. Maybe a little of both. A PEP, and an implementation and the ability to rationally debate, discuss and defend your proposal is what is needed to enact changes on policies, python-core or the standard library. >> The decision to move python's source control from SVN to mercurial was >> controversial enough; including 3 or more scm libraries into core >> would be an intractable uphill mountain of bike sheds. > > Not at all. > > It would be a very fair thing to do. Not to mention being > great for users. There should be one-- and preferably only one --obvious way to do it. > Anyway.. Maybe I do get what your talking about. Even if you do > talk with a strange accent. :-) My sense of humor has been disabled by repeated stunning at your hands. I admire your enthusiasm, even if I do think some of it is misplaced, or at guided into the proper channels at very least. Please, you seem to have the time and willingness to help, please go about this the right way. Discuss things on the proper lists, make concrete proposals. If you have have standard lib changes, discuss them on stdlib-sig, if you have ideas about python-the-language, or the interpreter, etc - please discuss it on python-ideas. jesse From ben+python at benfinney.id.au Wed Jan 20 05:16:25 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 15:16:25 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <87wrzdfm1y.fsf@benfinney.id.au> "David Lyon" writes: > Being honest, if wonderful libraries like Sphinx and Mercurial and Git > and BZR can't make it into the stdlib, then there is no hope for even > newer code to get in there. Those are applications, not libraries. Applications don't belong in the standard library. -- \ ?If you pick up a starving dog and make him prosperous, he will | `\ not bite you. This is the principal difference between a dog | _o__) and a man.? ?Mark Twain, _Pudd'n'head Wilson_ | Ben Finney From brian.curtin at gmail.com Wed Jan 20 05:19:47 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 19 Jan 2010 22:19:47 -0600 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: On Tue, Jan 19, 2010 at 21:43, David Lyon wrote: > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > I'm not entirely sure I see why the inclusion of a SCM into the stdlib is necessary. Just because pieces of software are mature and proven in their fields doesn't mean we should add them, or that them *not* being in the stdlib should be a basis for other projects making it in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Jan 20 05:26:44 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:26:44 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119232644.7fa25691@freewill> On Jan 20, 2010, at 02:43 PM, David Lyon wrote: >> On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: >> A SCM is not a "package management system". > >Exactly. It almost makes the need for a "package management system" >pretty much obsolete if you can update your code directly from >the developers sources. > >That's what all these SCMs provide. Plus it's addictive. It's >hard to go back to 'package' style technology once you have >all your code on an SCM based feed. Well... I'm not so sure. A package management system like apt does a /ton/ of additional bookkeeping and work to ensure a robust, highly consistent, functioning system. And while both Python and most Linux distributions have their own notion of "package management", they don't always play nicely together. Tarek and the distutils-sig's work is trying to make the world a better place by bridging this gap better, and there is code out there that makes it easier to say import a Python package from the Cheeseshop and .deb-ify it for use on Debian and Ubuntu. There's also work being done in Launchpad that will allow you to "build-from-branch" so that in a sense you could let a build farm take your Bazaar branches and automatically build the packages from them. I've strayed off-topic I suppose, but I see SCMs and package managers as complementary technologies that help with important parts of the process of delivering software to end-users, but I don't quite see how one can make the other obsolete. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 05:29:34 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 15:29:34 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> > On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: > > I'd be surprised if any of the big 3 DVCS developers would actually /want/ > their stuff in the stdlib. If they ask, they'll get told they're motorbike-shedding. "It's better if their users ask". So here I am as a user doing things the 'right' way. > Being in the stdlib has its advantages and > disadvantages. I think for rapidly developing technology, the > latter can actually outweigh the former. If it's about being able to do updates, then I think this resolves an old and circular argument. As the SCM implementation would, one would expect, to be able to update itself. Side benefits are that it can update everything else along with it at the same time. User Apps, Packages, whatever. It's even better having SCM in an Industrial/Scientific environment. Here's an example: - a machine breaks.. (I mean the software for/in it) - you fix the code, maybe on the spot - you commit and push back to the repository - your code gets checked in and run through the testbot and then you get blamed and have to do the whole thing again properly with a test case. Oh well.. Well anyway, whatever you guys might say, that's a whole lot more efficient than running back to the development machine and going through some obscure build and test and publish process to do a fix on a production machine. Point : The fact that SCMs are two way is great in a production environment. No packaging solution can come close. So why not have python SCMs included as batteries in python.. All these arguments I can take off to the stdlib list when I get the chance.. David From barry at python.org Wed Jan 20 05:50:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:50:36 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119235036.57f6e867@freewill> Okay, last follow up on this and then I'm going to bed. :) On Jan 20, 2010, at 03:29 PM, David Lyon wrote: >> On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: >> >> I'd be surprised if any of the big 3 DVCS developers would actually /want/ >> their stuff in the stdlib. > >If they ask, they'll get told they're motorbike-shedding. "It's better >if their users ask". So here I am as a user doing things the 'right' >way. Actually, you're not. It's not up to the Python community to initiate this. If you really want this, you should engage with the relevant DVCS communities and push them to request it. >Side benefits are that it can update everything else along >with it at the same time. User Apps, Packages, whatever. I get that. Heck, I still run one Gentoo server which I think is as close to the edge you're describing as I'm comfortable with. It's all great until the wheels come off and then it can take *days* to get a functioning system again. The big difference is that I rely on my DVCS to keep one small thing, or a few variants of the same thing, all sane. But I rely on my distribution vendor to keep a thousand complex, interdependent, interacting, sometimes conflicting things sane and working. >Point : The fact that SCMs are two way is great in > a production environment. No packaging solution > can come close. Try talking with some hard-core operations guys, the folks with the keys to the data centers, who work tireless, insanely hours keeping incredibly complex systems running with very little downtime. I think you'd get a different perspective to put it mildly. :) to-sleep-perchance-to-dream-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 06:10:15 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 16:10:15 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <87wrzdfm1y.fsf@benfinney.id.au> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> Message-ID: <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> > "David Lyon" writes: > >> Being honest, if wonderful libraries like Sphinx and Mercurial and Git >> and BZR can't make it into the stdlib, then there is no hope for even >> newer code to get in there. > > Those are applications, not libraries. Applications don't belong in the > standard library. Haha funny.. Well using that logic, distutils is an application.. Are you saying that distutils should be removed? That is most certainly an application. Lets not get too pedantic here. Mercurial and bzr have a built in API that can be called in a library like way. It's true they also have a command line interface in the same way that distutils does. I'm not saying anything negative about distutils. Given that Tarek has an upcoming Pycon presentation where the program talks about a distutils revamp. I'm hoping that he can find some young 20 yr olds and put a cool web interface on that thing. Given that there are empty sprints at pycon. It couldn't hurt to throw that challenge out. Anyway, we'll see.. David From ben+python at benfinney.id.au Wed Jan 20 06:32:10 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:32:10 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> <20100119235036.57f6e867@freewill> Message-ID: <87iqaxfijp.fsf@benfinney.id.au> Barry Warsaw writes: > On Jan 20, 2010, at 03:29 PM, David Lyon wrote: > >So here I am as a user doing things the 'right' way. > > Actually, you're not. It's not up to the Python community to initiate > this. If you really want this, you should engage with the relevant > DVCS communities and push them to request it. Where ?push? must be strictly limited by a continual awareness that the whole idea could just be bad. If you find yourself in a tiny minority pushing for a change, it *could* be that you have a great idea and the vast majority don't realise it yet. But you must be realistic about the likelihood that the change is a very *bad* idea, and frequently evaluate it for signs of that. -- \ ?I used to think that the brain was the most wonderful organ in | `\ my body. Then I realized who was telling me this.? ?Emo Philips | _o__) | Ben Finney From ben+python at benfinney.id.au Wed Jan 20 06:34:07 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:34:07 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> Message-ID: <87eillfigg.fsf@benfinney.id.au> "David Lyon" writes: > Well using that logic, distutils is an application.. Distutils is an application, the function of which is essential to allowing sane development of Python packages. It's a special case. We need to strictly limit the number of special cases, not gleefully add to them. -- \ ?I'd take the awe of understanding over the awe of ignorance | `\ any day.? ?Douglas Adams | _o__) | Ben Finney From matthieu.brucher at gmail.com Wed Jan 20 07:49:36 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 07:49:36 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. I think there are several points that make them not includable in Python: - git is not written in Python - bzr and mercurial have a life cycle much shorter than Python's, it's the same issue than with other libraries where another community develops them. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From david.lyon at pythontest.org Wed Jan 20 08:20:02 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 18:20:02 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Matthieu, >> I'd be happy to see bzr and mercurial and git all made it together >> into the stdlib for python 3. That would give a superb updating >> mechanism for python that would propel python well beyond >> the dinosaur badlands of CPAN and other languages. > > I think there are several points that make them not includable in Python: > - git is not written in Python > - bzr and mercurial have a life cycle much shorter than Python's, it's > the same issue than with other libraries where another community > develops them. That's only two points. :-) On 1; If that's true, I won't mention git again. On 2; Who knows what their life cycle is. CVS is pretty much dead, and svn looks like it is on the way out. I can't think of how anything could be better than mercurial or bzr but I know I will be proved wrong. At the end of the day, we are making a decision about whether the language is 'set-in-stone' or whether it is still evolving. To me, Python 1.x had it's own distinct "era", as has Python 2.x Hoping that the Python 3 era can be a little more flexible and perphaps "cleaner" than the 2.x era is all that I am thinking here. Have a nice day David From matthieu.brucher at gmail.com Wed Jan 20 09:05:19 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 09:05:19 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: > That's only two points. :-) In French, we say that several starts with 2 ;) > On 1; If that's true, I won't mention git again. I tis, you can check on the git repository (it's a mix of C, perl, shell scripts, Python, ...) > On 2; Who knows what their life cycle is. You can check on their websites, their cycles are far shorter than Python minor releases (several months vs several years). Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From abpillai at gmail.com Wed Jan 20 10:36:53 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Wed, 20 Jan 2010 15:06:53 +0530 Subject: [Python-Dev] E3 BEFEHLE In-Reply-To: References: Message-ID: <8548c5f31001200136r54bc9e2aw49485280864ecb2d@mail.gmail.com> On Sun, Jan 17, 2010 at 9:42 AM, Dj Gilcrease wrote: > 2010/1/16 Jack Diederich : > > Good lord, did this make it past other people's spam filters too? I > > especially liked the reference to "REGION -2,0 ; Rlyeh". Ph'nglui > > mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn to you too sir. > > Ya made it past mine too, it looks like a debug dump of a macro for a > some German game based either LOTR or Cthulhu > I initially thought it was a Python disassembler trace of some step of operations which failed, converted to German. In fact, I was looking for a question at the end regarding REPL. How very optimistic... > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Jan 20 11:22:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 20 Jan 2010 19:22:55 +0900 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <87aaw9gjnk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > (Besides, git in the stdlib doesn't make much sense :). "Dulwich." From solipsis at pitrou.net Wed Jan 20 12:28:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Jan 2010 11:28:16 +0000 (UTC) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon pythontest.org> writes: > > I think I'll have better chances with PEPs. > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > [snip] This is python-ideas material, can you take it there? Thank you. Antoine. From ncoghlan at gmail.com Wed Jan 20 13:16:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 22:16:34 +1000 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B56F422.5010607@gmail.com> David Lyon wrote: > On 2; Who knows what their life cycle is. CVS is pretty much > dead, and svn looks like it is on the way out. > I can't think of how anything could be better than > mercurial or bzr but I know I will be proved wrong. I believe you misunderstood what Matthieu meant by life cycle there: think "release cycle". If a project pushes out new releases significantly more often than every 18-24 months (as is currently true for all of the major SCM tools), then that fact alone makes it a very bad fit for the Python standard library. And centralised source control will be going strong for years. The DVCS approach may be great for the open source world, but the gains are far more limited in a closed source shop (especially a group writing internal corporate applications which doesn't need to keep many, if any, maintenance branches going). If we weren't dealing with 4 active branches, the DVCS discussion would have got a lot less traction with the core developers - aside from better handling of multiple lines of development, most of the benefits of the switch to a DVCS accrue to people without commit access to the SVN repository. Anyway, we've wandered far afield from legit python-dev topics now. Any further ideas about super_mega_easy_install functionality that can pull code from source control systems and build it rather than requiring prebuild source tarballs should be directed to python-ideas (they probably need to bake more even before they make an appearance on distutils-sig). Cheers, Nick. P.S. As Jesse said... your enthusiasm is great, but please don't assume that some inherent conservatism on the part of other developers is automatically evil or the result of a failure to see your point. A lot of people around the world rely on our stuff every day. We owe it to them to be measured in our actions and to put serious thought into any major changes or additions we make to the language and the standard library. For the current stage of its development, Python 3 is in a good place from our point of view - its major carrot has really always been the better Unicode support it offers, and the ever-increasing globalisation of the web will create more and more pressure pushing developers in that direction as the years go by. Sure, Python 3 cleans up assorted other things as well, but the change to the text processing model is the big one that is fundamentally incompatible with the architecture of the 2.x series. Compared to that change, everything else is just tinkering. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Fri Jan 1 16:07:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:07:20 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04D81D51@TK5EX14MBXC118.redmond.corp.microsoft.com> <94bdd2610912141458m52da21ear4970506a37214e6@mail.gmail.com> <4dab5f760912230700l38a597f7l855bb2304025d6d5@mail.gmail.com> Message-ID: <94bdd2611001010707h4043ff64x7476049e435852b@mail.gmail.com> 2009/12/23 Glyph Lefkowitz : [..] > 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". > > The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. ?In other words it's got "bin", "lib", "share", "etc", etc.. ?~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation. Right, good idea Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Fri Jan 1 16:32:27 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:32:27 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> Message-ID: <94bdd2611001010732o38a563bcu6d0cc9122ed5c88f@mail.gmail.com> On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziad? wrote: > Hello, > > A while ago I've proposed to refactor the APIs that provides access to > the installation paths and configuration variables at runtime into a > single module called "sysconfig", and make it easier for all > implementations to work with them. I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek From status at bugs.python.org Fri Jan 1 18:07:11 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Jan 2010 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100101170711.A5AAF78654@psf.upfronthosting.co.za> ACTIVITY SUMMARY (12/25/09 - 01/01/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2537 open (+22) / 16902 closed (+19) / 19439 total (+41) Open issues with patches: 1016 Average duration of open issues: 705 days. Median duration of open issues: 462 days. Open Issues Breakdown open 2504 (+22) pending 32 ( +0) Issues Created Or Reopened (42) _______________________________ doc: Code is not always colored 12/30/09 CLOSED http://bugs.python.org/issue7487 reopened flox documention buglet for PyBuffer 12/26/09 CLOSED http://bugs.python.org/issue7577 created ronaldoussoren easy Behavior of operations on a closed file object is not documented 12/26/09 http://bugs.python.org/issue7578 created blep Patch to add docstrings to msvcrt 12/26/09 http://bugs.python.org/issue7579 created brian.curtin patch distutils makefile from python.org installer doesn't work on OS 12/27/09 http://bugs.python.org/issue7580 created bskaplan incomplete doc of zlib 12/27/09 CLOSED http://bugs.python.org/issue7581 created coolwanglu [patch] diff.py to use iso timestamp 12/27/09 http://bugs.python.org/issue7582 created techtonik patch doctest should normalize tabs when comparing output 12/27/09 http://bugs.python.org/issue7583 created techtonik datetime.rfcformat() for Date and Time on the Internet 12/27/09 http://bugs.python.org/issue7584 created techtonik [patch] difflib should separate filename from timestamp with tab 12/27/09 http://bugs.python.org/issue7585 created techtonik patch Typo in collections documentation 12/28/09 CLOSED http://bugs.python.org/issue7586 created nneonneo Python 3.1.1 source build issues 12/28/09 CLOSED http://bugs.python.org/issue7587 created sharmaag unittest.TestCase.shortDescription isn't short anymore 12/28/09 http://bugs.python.org/issue7588 created exarkun setup.py shouldn't try to build nis module when nis headers aren 12/28/09 CLOSED http://bugs.python.org/issue7589 created Arfrever patch 'exceptions' module mentioned in documentation 12/28/09 CLOSED http://bugs.python.org/issue7590 created July test_get_platform fails on 3.1 12/28/09 http://bugs.python.org/issue7591 created pitrou buildbot ssl module documentation: SSLSocket.unwrap description shown twi 12/28/09 http://bugs.python.org/issue7592 created mnewman Computed-goto patch for RE engine 12/29/09 http://bugs.python.org/issue7593 created akuchling patch, needs review shlex refactoring 12/29/09 http://bugs.python.org/issue7594 created ferringb patch, needs review Doc typo for select.kevent() 12/29/09 CLOSED http://bugs.python.org/issue7595 created whit537 test_logging sometimes fails 12/29/09 CLOSED http://bugs.python.org/issue7596 created pitrou curses.use_env implementation error 12/30/09 http://bugs.python.org/issue7597 created kanru patch Cannot install, problems with assembly? 12/30/09 CLOSED http://bugs.python.org/issue7598 created sh.00 (c)ElementTree can produce invalid XML 12/30/09 http://bugs.python.org/issue7599 created jwilk interpreter crashes before start 12/30/09 CLOSED http://bugs.python.org/issue7600 created mhh91 Installation - 2 tests failed: test_cmd_line test_xmlrpc 12/30/09 CLOSED http://bugs.python.org/issue7601 created sadhak Doc: make clean and make update do not delete or update Doc/tool 12/30/09 CLOSED http://bugs.python.org/issue7602 created flox There is no 'seq' type 12/30/09 CLOSED http://bugs.python.org/issue7603 created tom66 delattr __slots__ inconsistancy 12/30/09 CLOSED http://bugs.python.org/issue7604 created ferringb test_cmd_line fails with non-ascii path 12/30/09 http://bugs.python.org/issue7605 created pitrou test_xmlrpc fails with non-ascii path 12/30/09 http://bugs.python.org/issue7606 created pitrou stringlib fastsearch could be improved on 64-bit builds 12/30/09 http://bugs.python.org/issue7607 created pitrou PyUnicode_FromFormatV handles %R and %S incorrectly. 12/30/09 CLOSED http://bugs.python.org/issue7608 created alexandre.vassalotti Add --with-system-expat option 12/31/09 CLOSED http://bugs.python.org/issue7609 created Arfrever patch Cannot use both read and readline method in same ZipExtFile obje 12/31/09 http://bugs.python.org/issue7610 created lucifer shlex not posix compliant when parsing "foo#bar" 12/31/09 http://bugs.python.org/issue7611 created jjdmol2 Fix "pass and object" typo in Library Reference / Built-in Types 12/31/09 CLOSED http://bugs.python.org/issue7612 created ygale patch [cppcheck] found a regression : Invalid number of character ((). 12/31/09 CLOSED http://bugs.python.org/issue7613 created ettl.martin patch Python 2.6.4 segfaults 12/31/09 CLOSED http://bugs.python.org/issue7614 created ttsiod unicode_escape codec does not escape quotes 01/01/10 http://bugs.python.org/issue7615 created rhansen patch test_memoryview test_setitem_writable failures with Intel ICC 01/01/10 http://bugs.python.org/issue7616 created ivank distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option 01/01/10 http://bugs.python.org/issue7617 created Arfrever patch Issues Now Closed (46) ______________________ True division of integers could be more accurate 715 days http://bugs.python.org/issue1811 mark.dickinson patch API to clear most free lists 690 days http://bugs.python.org/issue2019 amaury.forgeotdarc patch unit test UnicodeWarning 687 days http://bugs.python.org/issue2100 ezio.melotti test_disutils fails 568 days http://bugs.python.org/issue3054 ronaldoussoren patch test_formatdate_usegmt fails on non-englisch locale 451 days http://bugs.python.org/issue4046 r.david.murray "??" in u"foo" raises a misleading error 410 days http://bugs.python.org/issue4328 r.david.murray zipfile - add __exit__ attribute to make ZipFile object compatib 287 days http://bugs.python.org/issue5511 ezio.melotti patch test_urllib2 fails - urlopen error file not on local host 271 days http://bugs.python.org/issue5625 orsenthil Improve --with-dbmliborder option 170 days http://bugs.python.org/issue6491 benjamin.peterson patch Move the special-case for integer objects out of PyBytes_FromObj 141 days http://bugs.python.org/issue6687 alexandre.vassalotti patch, 26backport setup.py fails to find headers of system libffi 104 days http://bugs.python.org/issue6943 benjamin.peterson patch, easy The replacement suggested for callable(x) in py3k is not equival 92 days http://bugs.python.org/issue7006 benjamin.peterson patch C/API - Document exceptions 89 days http://bugs.python.org/issue7033 lekma patch subprocess.check_output: "docstring has inconsistent leading whi 35 days http://bugs.python.org/issue7381 georg.brandl patch optparse Documentation References Example Files that do not Exis 30 days http://bugs.python.org/issue7404 georg.brandl patch datetime.datetime.isoformat truncation problem 29 days http://bugs.python.org/issue7413 amaury.forgeotdarc patch, needs review doc: Code is not always colored 0 days http://bugs.python.org/issue7487 georg.brandl float('nan')**2 != nan. float('nan')**2 error 33 on windows 13 days http://bugs.python.org/issue7534 mark.dickinson patch If a generator raises TypeError when being unpacked, an unrelate 10 days http://bugs.python.org/issue7548 amaury.forgeotdarc ctypes doc improvement: c_char_p 6 days http://bugs.python.org/issue7569 georg.brandl Strange issue : cursor.commit() with sqlite 5 days http://bugs.python.org/issue7572 ghaering tes_math fails Mac OS X 10.4 due to OverflowError in test_mtestf 5 days http://bugs.python.org/issue7575 mark.dickinson patch documention buglet for PyBuffer 2 days http://bugs.python.org/issue7577 georg.brandl easy incomplete doc of zlib 0 days http://bugs.python.org/issue7581 amaury.forgeotdarc Typo in collections documentation 0 days http://bugs.python.org/issue7586 georg.brandl Python 3.1.1 source build issues 0 days http://bugs.python.org/issue7587 amaury.forgeotdarc setup.py shouldn't try to build nis module when nis headers aren 1 days http://bugs.python.org/issue7589 benjamin.peterson patch 'exceptions' module mentioned in documentation 1 days http://bugs.python.org/issue7590 georg.brandl Doc typo for select.kevent() 0 days http://bugs.python.org/issue7595 georg.brandl test_logging sometimes fails 1 days http://bugs.python.org/issue7596 ezio.melotti Cannot install, problems with assembly? 0 days http://bugs.python.org/issue7598 ezio.melotti interpreter crashes before start 0 days http://bugs.python.org/issue7600 r.david.murray Installation - 2 tests failed: test_cmd_line test_xmlrpc 1 days http://bugs.python.org/issue7601 ezio.melotti Doc: make clean and make update do not delete or update Doc/tool 0 days http://bugs.python.org/issue7602 georg.brandl There is no 'seq' type 0 days http://bugs.python.org/issue7603 benjamin.peterson delattr __slots__ inconsistancy 0 days http://bugs.python.org/issue7604 benjamin.peterson PyUnicode_FromFormatV handles %R and %S incorrectly. 0 days http://bugs.python.org/issue7608 alexandre.vassalotti Add --with-system-expat option 1 days http://bugs.python.org/issue7609 benjamin.peterson patch Fix "pass and object" typo in Library Reference / Built-in Types 0 days http://bugs.python.org/issue7612 ezio.melotti patch [cppcheck] found a regression : Invalid number of character ((). 0 days http://bugs.python.org/issue7613 ezio.melotti patch Python 2.6.4 segfaults 0 days http://bugs.python.org/issue7614 mark.dickinson Fwd: Debian Bug#42318: urllib.py has problems with malformed pro 3436 days http://bugs.python.org/issue210849 shinnosuke urllib / urllib2 should cache 301 redirections 2425 days http://bugs.python.org/issue735515 pitrou fast modular exponentiation 2084 days http://bugs.python.org/issue936813 mark.dickinson patch bdist_deb - Debian packager 316 days http://bugs.python.org/issue1054967 astraw patch Carbon.Scrap.PutScrapFlavor 987 days http://bugs.python.org/issue1700507 ronaldoussoren Top Issues Most Discussed (10) ______________________________ 11 Add Option to Bind to a Local IP Address in httplib.py 462 days open http://bugs.python.org/issue3972 8 fast modular exponentiation 2084 days closed http://bugs.python.org/issue936813 6 [patch] difflib should separate filename from timestamp with ta 5 days open http://bugs.python.org/issue7585 6 [patch] diff.py to use iso timestamp 5 days open http://bugs.python.org/issue7582 6 Implement fastsearch algorithm for rfind/rindex 24 days open http://bugs.python.org/issue7462 5 test_xmlrpc fails with non-ascii path 2 days open http://bugs.python.org/issue7606 5 test_logging sometimes fails 1 days closed http://bugs.python.org/issue7596 5 Patch to add docstrings to msvcrt 6 days open http://bugs.python.org/issue7579 5 Distutils-based installer does not detect 64bit versions of Pyt 126 days open http://bugs.python.org/issue6792 5 _sha256 et al. encode to UTF-8 by default 17 days open http://bugs.python.org/issue3745 From stefan_ml at behnel.de Sun Jan 3 09:11:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jan 2010 09:11:16 +0100 Subject: [Python-Dev] Providing support files to assist 3.x extension authors In-Reply-To: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> References: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> Message-ID: Case Vanhorsen, 20.12.2009 01:38: > When I ported gmpy (Python to GMP multiple precision library) to > Python 3.x, I began to use PyLong_AsLongAndOverflow frequently. I > found the code to slightly faster and cleaner than using PyLong_AsLong > and checking for overflow. You might want to look at the code Cython generates for integer type conversions. We use specialised coercion code that gets generated on-the-fly to convert Python long/int from and to all sorts of C integer types with compile time (portability) and runtime size/value checks. Depending on your needs, this may or may not be faster than the above C-API function. Stefan From david.lyon at preisshare.net Mon Jan 4 00:42:15 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 10:42:15 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> Message-ID: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> > On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: > > This new operator removes the ambiguity the original proposal had, > without making it more > complex for common use cases. So if you dislike it, you will need to > propose something > else that also fixes the ambiguity we had. Ok. > Environment markers >.. >Here are some example of fields using such markers: > >Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: [Windows] pywin32 1.0+ That's simpler, shorter, and less ambiguous. Easier to parse for package managers. David From python at mrabarnett.plus.com Mon Jan 4 01:14:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 04 Jan 2010 00:14:43 +0000 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4132F3.7020905@mrabarnett.plus.com> David Lyon wrote: >> On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: >> >> This new operator removes the ambiguity the original proposal had, >> without making it more >> complex for common use cases. So if you dislike it, you will need to >> propose something >> else that also fixes the ambiguity we had. > > Ok. > >> Environment markers >> .. >> Here are some example of fields using such markers: >> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. > 'win32' is more specific than 'Windows' and, to me, '1.0+' means '>=1.0', not '>1.0'. From martin at v.loewis.de Mon Jan 4 01:21:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jan 2010 01:21:53 +0100 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4134A1.5090203@v.loewis.de> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. Don't you want the PEP to complete? Why this bike-shedding? I can agree it's shorter. I can't agree that it's simpler, or less ambiguous. Regards, Martin From ssteinerx at gmail.com Mon Jan 4 01:29:14 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 3 Jan 2010 19:29:14 -0500 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: On Jan 3, 2010, at 7:21 PM, Martin v. L?wis wrote: >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Really.... Enough, already! S From david.lyon at preisshare.net Mon Jan 4 01:47:56 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 11:47:56 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: <1349.218.214.45.58.1262566076.squirrel@syd-srv02.ezyreg.com> Hi Martin, Happy New Year, >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Well, I'm just helping out by pointing out some simpler ways as Tarek asked me. I was only answering his question. I was out celebrating so it took longer to reply than normal. Bike-Shedding ? Me ? which bikeshed? wanting simple? Anyway, I'm just reading the PEPs and commenting. If there are some alterations that can be done, lets discuss them. > I can agree it's shorter. > .. Cool. What I'd really like is a 'Code-Repository:' keyword so that we can install programs/libraries directly into a system. I feel that this would really simplify the packaging landscape, making it much easier for the scientific community and others to do python software installs. This would allow us to perphaps have something even *more modern* than CPAN. So yes, getting PEP-345 right is important to me. Have a nice day. David From abpillai at gmail.com Mon Jan 4 10:08:05 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 4 Jan 2010 14:38:05 +0530 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: <8548c5f31001040108v635a78ech33521371c6c50e82@mail.gmail.com> On Sun, Dec 27, 2009 at 11:21 PM, anatoly techtonik wrote: > On Tue, Dec 15, 2009 at 12:37 AM, Steven Bethard > wrote: > > > > If you're only concerned about 2.X, then yes, optparse will *never* be > > removed from 2.X. There will be a deprecation note in the 2.X > > documentation but deprecation warnings will only be issued when the -3 > > flag is specified. Please see the "Deprecation of optparse" section of > > the PEP: > > > > http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Ref http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse . Considering that optparse will be deprecated like 5 years from now. I think this point is moot. The deprecation strategy IMHO is perhaps way too conservative. Maybe it should be deprecated faster ;) -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Mon Jan 4 10:32:10 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 4 Jan 2010 10:32:10 +0100 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: >> If you're only concerned about 2.X, then yes, optparse will *never* be >> removed from 2.X. There will be a deprecation note in the 2.X >> documentation but deprecation warnings will only be issued when the -3 >> flag is specified. Please see the "Deprecation of optparse" section of >> the PEP: >> >> http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Compare: > http://docs.python.org/library/optparse.html > http://argparse.googlecode.com/svn/tags/r101/doc/index.html > > argparse should be recommended as advanced and more featured variant > of optparse - that goes without saying, but forcing people to switch > and annoying them with deprecation messages brings only headache. As has been noted already nobody is forcing people to switch. Optparse will be available as a separate package and everybody will be free to install it and will not have any deprecation messages anywhere. > Just > like optparse is better getopt, the latter could also be useful for > people coming from other languages and porting their libraries to > Python. > > I would better concentrate on real code examples how argparse solves > problems and would really want to see > http://www.python.org/dev/peps/pep-0389/#out-of-scope-various-feature-requests > implemented until argparse enters phase where backward incompatible > changes in API are impossible. > > I would prefer to see PEP 389 as a document describing proposed > solutions to argument parsing problems rather than petition to replace > one library with another. So, it should display common argument > parsing scenarios (use cases) with examples that are also useful > recipes for documentation. I guess more than 90% people here doesn't > have time to read argparse methods descriptions to see what they could > be useful for. -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From olemis at gmail.com Mon Jan 4 15:24:05 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 4 Jan 2010 09:24:05 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> Message-ID: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: > Thanks for the quick response. > > On 30/12/2009, Benjamin Peterson wrote: >> >> When I made that change, I didn't know that the __unittest "hack" was >> being used elsewhere outside of unittest, so I felt fine replacing it >> with another. While I still consider it an implementation detail, I >> would be ok with exposing an "official" API for this. Perhaps >> __unittest_ignore_traceback? > > Well, bazaar has had the trick for a couple of years, and googling > around now turns up some other projects using it or thinking about it: > > > > Add `dutest` and probably `nose` [2]_ and ... > Reinstating the old implementation (with the same name) would mean > that existing code would keep working with Python 2.7 IOW ... if it is removed all the libraries will have to change and check for the Py version and ... (probably not a big deal depending on the details of the ?solution?) > but maybe a > discussion could start about a new, less hacky, way of doing the same > I am strongly -1 for modifying the classes in ?traditional? unittest module [2]_ (except that I am strongly +1 for the package structure WITHOUT TOUCHING anything else ...) , and the more I think about it I am more convinced ... but anyway, this not a big deal (and in the end what I think is not that relevant either ... :o). So ... pass > May not be worthwhile making life more complicated though, > there aren't *that* many unittest-extending projects. > But every library extending `unittest` will do it or otherwise not-so-useful error messages will be displayed ;o) PS: Happy New Year ! BTW .. [1] [feature] nosexml should omit trace frames where `__unittest... (http://code.google.com/p/python-nosexml/issues/detail?id=4#c0) .. [2] Assessment of unittest 2.7 API : new features and opinions (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free new ripit 1.3.3 Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/KFwyUTKH0vI/ From juanfhj at gmail.com Tue Jan 5 17:10:16 2010 From: juanfhj at gmail.com (Juan Fernando Herrera J.) Date: Tue, 5 Jan 2010 11:10:16 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility Message-ID: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Jan 5 17:50:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 05 Jan 2010 16:50:15 +0000 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B436DC7.8040605@voidspace.org.uk> On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way > major software hasn't been ported to it. I'm eager to use 3, but > paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me > if this has been suggested in the past. > I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well. I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter). All the best, Michael Foord > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Jan 5 18:21:31 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 5 Jan 2010 11:21:31 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion. I'm sure everyone out there would like to dive into supporting 3.x, but it's tough to prioritize when you are up to your eyeballs with 2.x bugs in your tracker. Look at Twisted ( http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3) -- over 1000 issues, ~5 developers -- 3.x support won't be here tomorrow, but it's on the horizon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 18:23:08 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2010 09:23:08 -0800 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B436DC7.8040605@voidspace.org.uk> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B436DC7.8040605@voidspace.org.uk> Message-ID: On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord wrote: > On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > > I don't know about other developers, but I certainly expected Python 3 > adoption to take a few years due to libraries only gradually making the > jump. I also expected there to be nervousness and pain around the migration > as well. > > I think in fact that libraries *are* migrating and there are lots of > encouraging signs. As a community we need to do all we can to promote Python > 3 adoption, but I don't think there is much reason to be worried (or > complacent for that matter). > Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Jan 5 18:52:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 18:52:20 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <319e029f1001050952o71cb29afh66445550beeb99c2@mail.gmail.com> On Tue, Jan 5, 2010 at 17:10, Juan Fernando Herrera J. wrote: > I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. Yes. Python 3 is not what you want to use today if you write applications. If you write libraries 2010 is the year to port, IMO. With some luck, 2011 will be the year to start porting applications properly. We'll see. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ianb at colorstudy.com Tue Jan 5 20:00:49 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 13:00:49 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin wrote: > On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > >> How about a new python 3 release with (possibly partial) backwards >> compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major >> software hasn't been ported to it. I'm eager to use 3, but paradoxically, >> the 3 release makes me rather stuck with 2.6. Excuse me if this has been >> suggested in the past. >> >> > The proper route to take, in my opinion, is to see what 2.x libraries you > are using that are not 3.x compatible, run 2to3 on them, then run their test > suite, and see where you get. Submit a patch or two to the library and see > what happens -- it at least gets the wheels in motion. > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 5 21:24:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 5 Jan 2010 15:24:45 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> On Jan 5, 2010, at 2:00 PM, Ian Bicking wrote: > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. It seems like this is a problem to be addressed, then. Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. > The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). No-op constructions like 'bytes("")' could help for older versions of Python, though. A very, very small runtime shim could provide support for these, if 2to3 could be told about it somehow. > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? Sort of like our own version of the #line directive? :) Seriously though, I find it hard to believe that this is a big problem. The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. > I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. This is almost certainly more of a pain than trying to trick 2to3 into doing the right thing. From regebro at gmail.com Tue Jan 5 21:57:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 21:57:35 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <319e029f1001051257k3827c52ahcd115b15d9a8ece7@mail.gmail.com> On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz wrote: > It seems like this is a problem to be addressed, then. ?Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x. >> The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated. > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 5 22:07:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 22:07:23 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B43AA0B.5030308@v.loewis.de> > It's not even that easy -- libraries can't apply patches for Python 3 > compatibility as they usually break Python 2 compatibility. Potentially > libraries could apply patches that make a codebase 2to3 ready, but from > what I've seen that's more black magic than straight forward updating, > as such patches have to trick 2to3 producing the output that is desired. I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x. > The only workable workflow I've seen people propose for maintaining a > single codebase with compatibility across both 2 and 3 is to use such > tricks, with aliases to suppress some 2to3 updates when they are > inappropriate I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way. > Also, running 2to3 on installation is kind of annoying, as you get > source that isn't itself the canonical source, so to fix bugs you have > to look at the installed source and trace it back to the bug in the > original source. That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin From martin at v.loewis.de Tue Jan 5 22:13:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jan 2010 22:13:07 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <4B43AB63.3000607@v.loewis.de> > No-op constructions like 'bytes("")' could help for older versions of > Python, though. A very, very small runtime shim could provide > support for these, if 2to3 could be told about it somehow. You actually don't *need* 2to3 support for that - bytes("") can work in either version: 2.x: def bytes(s): return s 3.x: def bytes(s) return s.encode("ascii") On top of that, you *could* as for bytes("") to be converted to b"" in 3.x - and it is indeed possible to tell 2to3 about that, and you don't even need to modify 2to3's source to do so: it can be part of your package. >> Also, running 2to3 on installation is kind of annoying, as you get >> source that isn't itself the canonical source, so to fix bugs you >> have to look at the installed source and trace it back to the bug >> in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in > .pycs rather than based on where the code was actually loaded in the > filesystem, couldn't 2to3 could do .pyc rewriting to point at the > original source? Sort of like our own version of the #line > directive? :) I think it could, but it would be fairly flaky as the pycs can get lost, and lose that information after regeneration. Also, it may be confusing in other scenarios. I'd rather have it generate separate mapper files, and change the traceback printing to consider these (as an option). > Seriously though, I find it hard to believe that this is a big > problem. The 3.x source looks pretty similar to the 2.x source, and > it's good to look at both if you're dealing with a 3.x issue. That's my experience as well - the only challenge is that you can't cut-n-paste the source URL into an editor. Regards, Martin From ianb at colorstudy.com Tue Jan 5 22:39:21 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 15:39:21 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43AA0B.5030308@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: On Tue, Jan 5, 2010 at 3:07 PM, "Martin v. L?wis" wrote: > > > It's not even that easy -- libraries can't apply patches for Python 3 > > compatibility as they usually break Python 2 compatibility. ?Potentially > > libraries could apply patches that make a codebase 2to3 ready, but from > > what I've seen that's more black magic than straight forward updating, > > as such patches have to trick 2to3 producing the output that is desired. > > I wouldn't qualify it in that way. It may be necessary occasionally to > trick 2to3, but that's really a bug in 2to3 which you should report, so > that trickery is then a work-around for a bug - something that you may > have to do with other API, as well. > > The "black magic" is really more in the parts that 2to3 doesn't touch > at all (because they are inherently not syntactic); these are the > problem areas Guido refers to. The "black magic" then is to make the > same code work unmodified for both 2.x and 3.x. Just to clarify, the black magic I'm referring to is things like: try: ?? ?unicode_ = unicode except NameError: ?? ?unicode_ = str and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). ?If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. ?It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. ?For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str. ?That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself. Also replying to Glyph: > > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? ?Sort of like our own version of the #line directive? :) > > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun. So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits. Which is a terribly crotchety of me. Sorry. -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From martin at v.loewis.de Tue Jan 5 23:23:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 23:23:53 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: <4B43BBF9.2000302@v.loewis.de> > Just to clarify, the black magic I'm referring to is things like: > > try: > unicode_ = unicode > except NameError: > unicode_ = str > > and some other aliases like this that are unambiguous and which 2to3 > won't touch (if you write them correctly). If the porting guide noted > all these tricks (of which several have been developed, and I'm only > vaguely aware of a few) that would be helpful. It's not a lot of > tricks, but the tricks are not obvious and 2to3 gets the translation > wrong pretty often without them. For instance, when I say str in > Python 2 I often means bytes, unsurprisingly, but 2to3 translates both > str and unicode to str. No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3? > That *nothing* translates to bytes by default > (AFAIK) means that people must either be living in a bytes-free world > (which sure, lots of code does) or they are using tricks not included > in 2to3 itself. By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3). > Since 2to3 maintains line numbers yes, it wouldn't be that bad. But > then I don't currently develop any code that is "installed", I only > develop code that is directly from a source code checkout, and where > the checkout is put on the path. I guess I could have something that > automatically builds the code on every edit, and that's not > infeasible. It's just not fun. Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it. > So long as I have to support Python 2 > (which is like forever) then adding Python 3 only makes development > that much more complicated and much less fun, with no concrete > benefits. I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin From ziade.tarek at gmail.com Wed Jan 6 01:08:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 01:08:34 +0100 Subject: [Python-Dev] PEP 386 and PEP 345 Message-ID: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Hi, I think we've reached a consensus on those two PEPs. Although, there's one last point that was forgotten in the discussions : I've introduced "rc" in the pre-releases markers, so PEP 386 is compatible with Python's own version scheme. "rc" comes right after "c" in the sorting. It's slightly redundant with the "c" marker but I don't think this really matters as long as consumers know how to order them (a < b < c < rc). I have also stated that "c" is the preferred marker for third party projects, from PEP 386 point of view. Is there anything else I can do to make those two PEPs accepted ? Regards Tarek -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Wed Jan 6 01:26:34 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:26:34 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43BBF9.2000302@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> <4B43BBF9.2000302@v.loewis.de> Message-ID: <1322.218.214.45.58.1262737594.squirrel@syd-srv02.ezyreg.com> Hi Martin, > ... but in the long run, starting now will have saved you a lot of > pressure in the long run, and stop users from switching away from > your packages because of lack of Python 3 support. In a production situation it works the other way around. If there's an application that requires twisted (or whatever package) then most people would use the appropriate interpreter to match the library. Since you guys all did your jobs so well :-) doing so is painless. Because there is so much "comfort" with the existing situation it makes it an effort for people to move to a different lounge chair. Namely python 3. I'd suggest that moving the package set (pypi) to python 3 could be kicked along with the help of some automated tools. I don't know what tools you guys have got. But I am very sure that if code analysis was provided to package developers on python 3 (so they don't have to run it themselves), then it would be like an even bigger tv screen in a bigger lounge room and it would assist in drawing them over. David From david.lyon at preisshare.net Wed Jan 6 01:36:24 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:36:24 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <1337.218.214.45.58.1262738184.squirrel@syd-srv02.ezyreg.com> Hi Ian, > The only workable workflow I've seen people propose for maintaining a > single > codebase with compatibility across both 2 and 3 is to use such tricks, > with > aliases to suppress some 2to3 updates when they are inappropriate, so that > you can run 2to3 on install and have a single canonical Python 2 source. > Python 2.7 won't help much (even though it is trying) as the introduction > of non-ambiguous constructions like b"" aren't compatible with previous > versions of Python and so can't be used in many libraries (support at > least > back to Python 2.5 is the norm for most libraries, I think). > > Also, running 2to3 on installation is kind of annoying, as you get source > that isn't itself the canonical source, so to fix bugs you have to look at > the installed source and trace it back to the bug in the original source. > > I suspect a reasonable workflow might be possible with hg and maybe patch > queues, but I don't feel familiar enough with those tools to map that out. That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David From david.lyon at preisshare.net Wed Jan 6 03:01:22 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 13:01:22 +1100 (EST) Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: <1617.218.214.45.58.1262743282.squirrel@syd-srv02.ezyreg.com> > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? Tarek, Given that I helped out so much last year with the PEP in discussing different options, even if they weren't accepted, I really feel that it is unfair if my name isn't mentioned. It was a huge time sacrifice on my part. For example, even if I only managed to explain the version numbering and clarify how that worked. It did take me some time to do that. What I did do however, was spend a lot of time with the multiplatform "Markers". I still think that two short weeks more of "discussion" could resolve some issues. That discussion went for 4 months on distutils-ml. Look, major issues aside, can you make the following concessions on PEP-345 which I only feel will help it, namely: 1) Source-Repository: specify a code repository to install from 2) Streamline Requires-Python: by implementing "Markers" as noted by the PEP. A marker being something like "Requires-Python(windows): lxml". Otherwise remove the word marker from the PEP and just replace with "metacode". Markers are what were discussed on distutils-ml. Metacode is what is described in the PEP. 3) Remove the inconsistency and platform ambiguities. Especially for windows users. For example, "win32" is extremely confusing for windows users right now. As more and more systems now are 64 bit. Use the platform module, instead of the sys module for constants. I'll post to distutils-ml on this. I am certainly not trying to hold this PEP up, and I apologise on my part for my late attention. I will post to distutils-ml on these and i promise to keep my comments unheated and unwitty. Having said that, PEP-345 is *super-important*. A week or two or three more discussion and the issues can be resolved. We all just want to focus on being productive. It would be a great accomplishment for you to get PEP-345 approved and likewise for me getting mentioned even in a minor role as helping out on a PEP. So I'm hoping that you can make a few last minute concessions meaning that we can all happily go on our way in 2010. Regards David From brett at python.org Wed Jan 6 06:20:30 2010 From: brett at python.org (Brett Cannon) Date: Tue, 5 Jan 2010 21:20:30 -0800 Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 16:08, Tarek Ziad? wrote: > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? > As you said, consensus has been reached, so just Guido's BDFL stamp of approval is all I can think of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Jan 6 12:19:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:19:45 +0000 Subject: [Python-Dev] bug triage Message-ID: <4B4471D1.9020707@simplistix.co.uk> Hi All, Is there a high volume of incoming bugs to the Python tracker? If so, I'd like to help with triaging. I think I have all the necessary access, what I'm missing is the knowledge of how to set myself up to get notifications of new bugs... How do I do that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at voidspace.org.uk Wed Jan 6 12:24:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 06 Jan 2010 11:24:37 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: <4B4472F5.8000709@voidspace.org.uk> On 06/01/2010 11:19, Chris Withers wrote: > Hi All, > > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the > necessary access, what I'm missing is the knowledge of how to set > myself up to get notifications of new bugs... > > How do I do that? > Bug triaging is one of Python's "big needs" and anything you do to help on this score would be much appreciated. Particularly reviewing new and outstanding issues. I assumed there would be RSS feeds for bug tracker activity but can't easily find these on the tracker. There is a bot that posts activity to #python-dev, so there must be some way of getting this information. All the best, Michael > cheers, > > Chris > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Jan 6 12:25:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 11:25:16 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: Hi Chris, > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Do you really want to get such notifications? There may be a lot of them. If you want however, you can join #python-dev on IRC (irc.freenode.net) where there's a bot which posts updates of all bugs on the tracker. There's usually not a lot of discussion going on so you probably won't feel flooded. In addition to bug triage, what is needed is reviewing of existing patches, as well as writing patches for issues which haven't been addressed yet :-) Regards Antoine. From chris at simplistix.co.uk Wed Jan 6 12:30:40 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:30:40 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B447460.7080100@simplistix.co.uk> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. Yeah, email-out is what I'm really after... I have it for my own Roundup instance so it can't be that hard to do ;-) Roch?, you guys host the bug tracker, right? Is there email-out set up for it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Jan 6 12:37:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 21:37:23 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B4475F3.5040406@gmail.com> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. I'm pretty sure the bugs list is still the primary spooled notification mechanism: http://mail.python.org/mailman/listinfo/python-bugs-list There are also the weekly tracker activity summaries that are posted here to python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From chris at simplistix.co.uk Wed Jan 6 12:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:41:28 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <4B4476E8.8050709@simplistix.co.uk> Nick Coghlan wrote: > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list That's what I was after, thanks! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From facundobatista at gmail.com Wed Jan 6 13:03:08 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 6 Jan 2010 09:03:08 -0300 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: On Wed, Jan 6, 2010 at 8:19 AM, Chris Withers wrote: > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Not notifications, but maybe a way to have a higher look of them for easy selection: http://www.taniquetil.com.ar/cgi-bin/pytickets.py Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From ziade.tarek at gmail.com Wed Jan 6 13:29:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:29:58 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord wrote: > On 06/01/2010 11:19, Chris Withers wrote: >> >> Hi All, >> >> Is there a high volume of incoming bugs to the Python tracker? >> If so, I'd like to help with triaging. I think I have all the necessary >> access, what I'm missing is the knowledge of how to set myself up to get >> notifications of new bugs... >> >> How do I do that? >> > > Bug triaging is one of Python's "big needs" and anything you do to help on > this score would be much appreciated. Particularly reviewing new and > outstanding issues. Another useful triage I think, is to review the oldest bugs (some of them are > 5 years) and remove the ones that are not relevant anymore, or duplicate with newer entries. Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Wed Jan 6 13:31:08 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 12:31:08 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B44828C.4070201@simplistix.co.uk> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with someone as a paired task for those 2 days... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Jan 6 13:37:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:37:55 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B44828C.4070201@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B44828C.4070201@simplistix.co.uk> Message-ID: <94bdd2611001060437s2d0242aembc669c3263773fea@mail.gmail.com> On Wed, Jan 6, 2010 at 1:31 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Another useful triage I think, is to review the oldest bugs (some of >> them are > 5 years) >> and remove the ones that are not relevant anymore, or duplicate with >> newer entries. > > I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with > someone as a paired task for those 2 days... I'll be doing Distutils stuff but I can probably help around a bit in that task: Distutils have quite a few old issues so I can tackle those From roche at upfrontsystems.co.za Wed Jan 6 13:32:39 2010 From: roche at upfrontsystems.co.za (=?ISO-8859-1?Q?Roch=E9?= Compaan) Date: Wed, 06 Jan 2010 14:32:39 +0200 Subject: [Python-Dev] bug triage In-Reply-To: <4B447460.7080100@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B447460.7080100@simplistix.co.uk> Message-ID: <1262781159.2836.218.camel@didi> On Wed, 2010-01-06 at 11:30 +0000, Chris Withers wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > Yeah, email-out is what I'm really after... I have it for my own Roundup > instance so it can't be that hard to do ;-) > > Roch?, you guys host the bug tracker, right? Is there email-out set up > for it? We do, but we don't administer it. There are a few administrators taking care of it and you should be able to reach them by logging your request here: http://psf.upfronthosting.co.za/roundup/meta/ or post it to the Infrastructure mailing list: infrastructure at python.org -- Roch? Compaan Upfront Systems http://www.upfrontsystems.co.za From ncoghlan at gmail.com Wed Jan 6 13:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 22:57:24 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B4488B4.2070208@gmail.com> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I believe someone (Daniel Diniz, maybe?) did do a pass over those some time in the last 12 months, so most of the obviously irrelevant ones that are that old should already be gone. Not to say it isn't worth doing another pass, just saying not to get disheartened if there aren't many that can be readily closed. There are at least a few still kicking around just because they're difficult to deal with (there's an ancient one to do with one of the ways circular imports can fail that I occasionally go back and reread before moving on to something more tractable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Wed Jan 6 14:43:17 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 08:43:17 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <4B4476E8.8050709@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> <4B4476E8.8050709@simplistix.co.uk> Message-ID: <20100106134317.3EF141FB5A6@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:41:28 +0000, Chris Withers wrote: > Nick Coghlan wrote: > > I'm pretty sure the bugs list is still the primary spooled notification > > mechanism: > > http://mail.python.org/mailman/listinfo/python-bugs-list > > That's what I was after, thanks! Just for completeness, there's also new-bugs-announce if you want just *new* bug notification. That's more for people who want to watch for bugs they want to become nosy on, though; if you are doing triage python-bugs-list is what you want. Please also read http://www.python.org/dev/workflow/ if you haven't already. Thanks for being willing to chip in! --David From brian.curtin at gmail.com Wed Jan 6 15:57:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 08:57:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4488B4.2070208@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > I believe someone (Daniel Diniz, maybe?) did do a pass over those some > time in the last 12 months, so most of the obviously irrelevant ones > that are that old should already be gone. Not to say it isn't worth > doing another pass, just saying not to get disheartened if there aren't > many that can be readily closed. > > There are at least a few still kicking around just because they're > difficult to deal with (there's an ancient one to do with one of the > ways circular imports can fail that I occasionally go back and reread > before moving on to something more tractable). > > Cheers, > Nick. > On the topic of bugs that can be readily closed (literally), I've recently come across a number of issues which appear to be sitting in a patch or review stage, but their patches have been committed and the issue remains open. What is the best course of action there? I'd just go ahead and close the issue myself but I don't have tracker privileges. I'm willing to help out with another Daniel Diniz-esque triage sweep if that would help. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Wed Jan 6 16:14:20 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 6 Jan 2010 10:14:20 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <7FCF8B19-3465-4392-80CC-BEAEBD926ECA@gmail.com> On Jan 6, 2010, at 7:29 AM, Tarek Ziad? wrote: > On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord > wrote: >> On 06/01/2010 11:19, Chris Withers wrote: >>> >>> Hi All, >>> >>> Is there a high volume of incoming bugs to the Python tracker? >>> If so, I'd like to help with triaging. I think I have all the necessary >>> access, what I'm missing is the knowledge of how to set myself up to get >>> notifications of new bugs... >>> >>> How do I do that? >>> >> >> Bug triaging is one of Python's "big needs" and anything you do to help on >> this score would be much appreciated. Particularly reviewing new and >> outstanding issues. > > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I was actually thinking about that the other day when I saw that the average age of bugs on the Python tracker was at some hideously large 3 digit number. The 'success' statistic would be to bring that down below, say, 100. S From skip at pobox.com Wed Jan 6 17:38:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 6 Jan 2010 10:38:13 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <19268.48245.616046.874126@montanaro.dyndns.org> >>>>> "Nick" == Nick Coghlan writes: Nick> I'm pretty sure the bugs list is still the primary spooled Nick> notification mechanism: Nick> http://mail.python.org/mailman/listinfo/python-bugs-list Actually, there is a new-bugs-announce list: http://mail.python.org/mailman/listinfo/new-bugs-announce Skip From solipsis at pitrou.net Wed Jan 6 18:56:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 17:56:49 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > On the topic of bugs that can be readily closed (literally), I've > recently come across a number of issues which appear to be sitting in a > patch or review stage, but their patches have been committed and the > issue remains open. What is the best course of action there? Post a message on the issue asking for info. From solipsis at pitrou.net Wed Jan 6 19:09:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 18:09:39 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > > On the topic of bugs that can be readily closed (literally), I've > > recently come across a number of issues which appear to be sitting in a > > patch or review stage, but their patches have been committed and the > > issue remains open. What is the best course of action there? > > Post a message on the issue asking for info. Ok, I realize my answer might have been a bit terse :-) The patch might be waiting to be merged in all development branches, or it may not totally resolve the issue, or perhaps documentation needs to be updated, or perhaps it is pending a verdict from the buildbots, etc. You can't deduce that the issue is completely fixed from the simple fact that something has been committed. Regards Antoine Pitrou. From brett at python.org Wed Jan 6 20:03:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 11:03:32 -0800 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > >> I believe someone (Daniel Diniz, maybe?) did do a pass over those some >> time in the last 12 months, so most of the obviously irrelevant ones >> that are that old should already be gone. Not to say it isn't worth >> doing another pass, just saying not to get disheartened if there aren't >> many that can be readily closed. >> >> There are at least a few still kicking around just because they're >> difficult to deal with (there's an ancient one to do with one of the >> ways circular imports can fail that I occasionally go back and reread >> before moving on to something more tractable). >> >> Cheers, >> Nick. >> > > On the topic of bugs that can be readily closed (literally), I've recently > come across a number of issues which appear to be sitting in a patch or > review stage, but their patches have been committed and the issue remains > open. What is the best course of action there? I'd just go ahead and close > the issue myself but I don't have tracker privileges. > > If a core developer is willing to step forward and vouch for you to get tracker privileges then I will give them to you. We are trying to give out tracker privs w/ less time than required to get commit privileges. So as long as you have helped out on a few issues in a positive and correct way that should be enough to get one of the regulars who perform triage to notice. -Brett I'm willing to help out with another Daniel Diniz-esque triage sweep if that > would help. > > Brian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Wed Jan 6 21:41:05 2010 From: nad at acm.org (Ned Deily) Date: Wed, 06 Jan 2010 12:41:05 -0800 Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: In article <4B4475F3.5040406 at gmail.com>, Nick Coghlan wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list Also, that mailing list (along with most python development related mailing lists) is mirrored at gmane.org which means it can also be obtained via a newsreader (NNTP) or various RSS feeds. http://dir.gmane.org/gmane.comp.python.bugs -- Ned Deily, nad at acm.org From nick.bastin at gmail.com Wed Jan 6 22:14:54 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:14:54 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? Message-ID: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> (This may occur on more platforms - I can test on more unix platforms if the consensus is this is an actual problem and I'm not just a nut) On freebsd5, if you do a simple ./configure --enable-shared in current (2.7) trunk, your python shared library will build properly, but all modules will fail to find the shared library and thus fail to build: gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.7/u1/Python/Python-2.7a1/Modules/_struct.o -L/u1/tmp/python2.7a1/lib -L/usr/local/lib -lpython2.7 -o build/lib.freebsd-5.3-RELEASE-i386-2.7/_struct.so /usr/bin/ld: cannot find -lpython2.7 building '_ctypes_test' extension ... This of course is because libpython2.7.so is in the current directory and not (yet) installed in /usr/local/lib. I've made a very simple fix for this problem that works, but at least to me smells a bit funny, which is to modify setup.py to add the following to detect_modules(): # If we did --enable-shared, we need to be able to find the library # we just built in order to build the modules. if platform == 'freebsd5': add_dir_to_list(self.compiler_obj.library_dirs, '.') Which brings me to a few questions: a) Does this seem like a real problem, or am I missing something obvious? b) Does this fix seem like the sensible thing to do? (it seems at least that we ought to check that the user configured --enable-shared and only set -L. in that case, if that's possible) Setting --enable-shared when you actually have a libpython2.7.so in /usr/local/lib (or whatever --prefix you've selected) is possibly even more dangerous, because it may succeed in linking against a differently-built library than what you intended. -- Nick From nick.bastin at gmail.com Wed Jan 6 22:39:17 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:39:17 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <66d0a6e11001061339t38202b70mca1091e1926ecf4b@mail.gmail.com> On Wed, Jan 6, 2010 at 16:14, Nicholas Bastin wrote: > This of course is because libpython2.7.so is in the current directory > and not (yet) installed in /usr/local/lib. One minor correction - as you could see from the compile line, the actual --prefix in this case is /u1/tmp/python2.7a1, but the libraries obviously aren't installed there yet either. Perhaps a better fix than setting -L. would be to put the shared library in build/lib.freebsd-5.3-RELEASE-i386-2.7 and add that to the library path for the linker (the build creates this directory, but installs nothing in it). -- Nick From martin at v.loewis.de Wed Jan 6 23:21:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jan 2010 23:21:44 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <4B450CF8.8090009@v.loewis.de> > b) Does this fix seem like the sensible thing to do? No. Linking in setup.py should use the same options as if the module was built as *shared* through Modules/Setup, which, IIUC, should use BLDLIBRARY. Regards, Martin From nick.bastin at gmail.com Thu Jan 7 01:08:13 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 19:08:13 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B450CF8.8090009@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> Message-ID: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> On Wed, Jan 6, 2010 at 17:21, "Martin v. L?wis" wrote: >> b) Does this fix seem like the sensible thing to do? > > No. Linking in setup.py should use the same options as if the module > was built as *shared* through Modules/Setup, which, IIUC, should use > BLDLIBRARY. Thanks for that pointer, that makes much more sense. Indeed, BLDLIBRARY on FreeBSD* is set to '-L. -lpython$(VERSION)' if you set --enable-shared, but somehow that piece of information doesn't propagate into the module build. More investigation to be done... -- Nick From rdmurray at bitdance.com Thu Jan 7 02:22:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 20:22:42 -0500 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > > On the topic of bugs that can be readily closed (literally), I've recently > > come across a number of issues which appear to be sitting in a patch or > > review stage, but their patches have been committed and the issue remains > > open. What is the best course of action there? I'd just go ahead and close > > the issue myself but I don't have tracker privileges. > > > > > If a core developer is willing to step forward and vouch for you to get > tracker privileges then I will give them to you. We are trying to give out > tracker privs w/ less time than required to get commit privileges. So as > long as you have helped out on a few issues in a positive and correct way > that should be enough to get one of the regulars who perform triage to > notice. > > -Brett I've done a quick scan of issues Brian is nosy on to refresh my memory, and I'd say he's definitely been making positive contributions. I'm willing to volunteer to keep an eye on his triage work for a while if you grant him tracker privs. Brian, I assume you'll be cognizant of Antoine's advice about making sure a bug really should be closed before closing it :) Hanging out in #python-dev on freenode while working on issues can be helpful, as well, since you can quickly ask whoever is there for second opinions on particular bugs. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Thu Jan 7 02:28:26 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 17:28:26 -0800 Subject: [Python-Dev] bug triage In-Reply-To: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > > On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > > On Wed, Jan 6, 2010 at 06:57, Brian Curtin > wrote: > > > On the topic of bugs that can be readily closed (literally), I've > recently > > > come across a number of issues which appear to be sitting in a patch or > > > review stage, but their patches have been committed and the issue > remains > > > open. What is the best course of action there? I'd just go ahead and > close > > > the issue myself but I don't have tracker privileges. > > > > > > > > If a core developer is willing to step forward and vouch for you to get > > tracker privileges then I will give them to you. We are trying to give > out > > tracker privs w/ less time than required to get commit privileges. So as > > long as you have helped out on a few issues in a positive and correct way > > that should be enough to get one of the regulars who perform triage to > > notice. > > > > -Brett > > I've done a quick scan of issues Brian is nosy on to refresh my > memory, and I'd say he's definitely been making positive contributions. > I'm willing to volunteer to keep an eye on his triage work for a while > if you grant him tracker privs. > > Done for the username brian.curtin (email doesn't match the one Brian emailed from so do let me know, Brian if this is the right username). Welcome aboard! -Brett > Brian, I assume you'll be cognizant of Antoine's advice about making > sure a bug really should be closed before closing it :) Hanging out in > #python-dev on freenode while working on issues can be helpful, as well, > since you can quickly ask whoever is there for second opinions on > particular bugs. > > -- > R. David Murray www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Jan 7 02:59:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 19:59:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 19:28, Brett Cannon wrote: > > > On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > >> >> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: >> > On Wed, Jan 6, 2010 at 06:57, Brian Curtin >> wrote: >> > > On the topic of bugs that can be readily closed (literally), I've >> recently >> > > come across a number of issues which appear to be sitting in a patch >> or >> > > review stage, but their patches have been committed and the issue >> remains >> > > open. What is the best course of action there? I'd just go ahead and >> close >> > > the issue myself but I don't have tracker privileges. >> > > >> > > >> > If a core developer is willing to step forward and vouch for you to get >> > tracker privileges then I will give them to you. We are trying to give >> out >> > tracker privs w/ less time than required to get commit privileges. So as >> > long as you have helped out on a few issues in a positive and correct >> way >> > that should be enough to get one of the regulars who perform triage to >> > notice. >> > >> > -Brett >> >> I've done a quick scan of issues Brian is nosy on to refresh my >> memory, and I'd say he's definitely been making positive contributions. >> I'm willing to volunteer to keep an eye on his triage work for a while >> if you grant him tracker privs. >> >> > Done for the username brian.curtin (email doesn't match the one Brian > emailed from so do let me know, Brian if this is the right username). > Welcome aboard! > > Yep, that's the one. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Jan 7 04:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 07 Jan 2010 03:07:56 +0000 Subject: [Python-Dev] GIL required for _all_ Python calls? Message-ID: <4B45500C.8090905@mrabarnett.plus.com> Hi, I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? Or is it just a case of checking the source files for mentions of the GIL? The header file for PyList_New, for example, doesn't mention it! Thanks From john.arbash.meinel at gmail.com Thu Jan 7 04:25:48 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 06 Jan 2010 21:25:48 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B45543C.2090200@gmail.com> MRAB wrote: > Hi, > > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? Or is it just a case of checking the source > files for mentions of the GIL? The header file for PyList_New, for > example, doesn't mention it! > > Thanks Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may get concurrent updating of the value, and then the final value is wrong. (two threads do 5+1 getting 6, rather than 7, and when the decref, you end up at 4 rather than back at 5). AFAIK, the only things that don't require the GIL are macro functions, like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for example, will be increfing and setting the exception state, so certainly needs the GIL to be held. John =:-> From benjamin at python.org Thu Jan 7 04:32:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jan 2010 21:32:17 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45543C.2090200@gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> Message-ID: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> 2010/1/6 John Arbash Meinel : > Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may > get concurrent updating of the value, and then the final value is wrong. > (two threads do 5+1 getting 6, rather than 7, and when the decref, you > end up at 4 rather than back at 5). Correct. > > AFAIK, the only things that don't require the GIL are macro functions, > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > example, will be increfing and setting the exception state, so certainly > needs the GIL to be held. As a general rule, I would say, no Py* macros are safe without the gil either (the exception being Py_END_ALLOW_THREADS), since they mutate Python objects which must be protected. -- Regards, Benjamin From guido at python.org Thu Jan 7 05:29:03 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Jan 2010 20:29:03 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: On Wed, Jan 6, 2010 at 7:32 PM, Benjamin Peterson wrote: > 2010/1/6 John Arbash Meinel : > > AFAIK, the only things that don't require the GIL are macro functions, > > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > > example, will be increfing and setting the exception state, so certainly > > needs the GIL to be held. > > As a general rule, I would say, no Py* macros are safe without the gil > either (the exception being Py_END_ALLOW_THREADS), since they mutate > Python objects which must be protected. That's keeping it on the safe side, since there are some macros like PyString_AS_STRING() that are also safe, *if* you are owning at least one reference to the string object. At the same time, "no Py* macros" is not quite strong enough, since if you called PyString_AS_STRING() before releasing the GIL but you don't own a reference to the string object, the string might be deallocated behind your back by another thread. A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). -- --Guido van Rossum (python.org/~guido) From yoann.padioleau at facebook.com Thu Jan 7 08:24:42 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Wed, 6 Jan 2010 23:24:42 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt Message-ID: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a file called ast.txt. Where does this file come from ? Is it generated from Python.asdl ? From johan.gill at agama.tv Thu Jan 7 10:46:52 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 10:46:52 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. Message-ID: <4B45AD8C.5000405@agama.tv> Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before, so this is probably an already solved problem. Regards Johan Gill Agama Technologies From regebro at gmail.com Thu Jan 7 12:12:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 12:12:17 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45AD8C.5000405@agama.tv> References: <4B45AD8C.5000405@agama.tv> Message-ID: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I imagine that other companies have contributed before, so this is > probably an already solved problem. I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hodgestar+pythondev at gmail.com Thu Jan 7 12:28:01 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 7 Jan 2010 13:28:01 +0200 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon From swamy.sangamesh at gmail.com Thu Jan 7 11:46:59 2010 From: swamy.sangamesh at gmail.com (swamy sangamesh) Date: Thu, 7 Jan 2010 16:16:59 +0530 Subject: [Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9 Message-ID: Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have attached the full test case result. If i change the type c_int and c_short to c_unit and c_ushort of class "BITS(Structure)" in file test_bitfields.py then no failures are seen. Has anyone faced the similar issue or any help is appreciated. Thanks, Sangamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ctype-testcases Type: application/octet-stream Size: 22996 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 7 13:23:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 07 Jan 2010 22:23:11 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: <4B45D22F.40404@gmail.com> Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls we need to think >> about? I imagine that other companies have contributed before, so this is >> probably an already solved problem. > > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license without requesting additional permission from the copyright holder. For corporate contributions, I believe the contributor agreement needs to be signed by an authorised agent of the company - the place to check that would probably be psf at python.org (that's the email address for the PSF board). Assuming the subject line relates to the code that you would like to contribute though, that particular change is unlikely to happen - 2.6 is in maintenance mode and changing RLock from a Python implementation to the faster C one is solidly in new feature territory. Although a backport of the 3.2 C RLock implementation to 2.7 could be useful, I doubt that backporting code provided by an existing committer would be the subject of this query :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Thu Jan 7 14:11:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:11:27 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. Even if the contributed code as in this case is a method in an existing file? How does that work, how do they keep ownership of one method in the threading.py method? :-) > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) Ah. I probably misunderstood what the suggested contribution was. Maybe it was a separate file, which I didn't get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Thu Jan 7 14:15:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 07 Jan 2010 13:15:09 +0000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> Message-ID: <4B45DE5D.3030104@voidspace.org.uk> On 07/01/2010 13:11, Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > >> As Simon pointed out, while some organisations do work that way, the PSF >> isn't one of them. >> >> The PSF only requires that the code be contributed under a license that >> then allows us to turn around and redistribute it under a different open >> source license without requesting additional permission from the >> copyright holder. >> > Even if the contributed code as in this case is a method in an > existing file? How does that work, how do they keep ownership of one > method in the threading.py method? :-) > > When contributing code to Python all work remains copyright the original author. The combined work is copyright *everyone*. The PSF has a license to distribute it, which is all that is important. How you meaningfully exercise your ownership over chunks of code is left for the reader to determine... (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From stefan_ml at behnel.de Thu Jan 7 14:30:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:30:16 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: Guido van Rossum, 07.01.2010 05:29: > A better rule would be "you may access the memory buffer in a PyString > or PyUnicode object with the GIL released as long as you own a > reference to the string object." Everything else is out of bounds (or > not worth the bother). Is that a "yes" regarding the OP's original question about releasing the GIL during regexp searches? Stefan From regebro at gmail.com Thu Jan 7 14:36:42 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:36:42 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DE5D.3030104@voidspace.org.uk> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> <4B45DE5D.3030104@voidspace.org.uk> Message-ID: <319e029f1001070536t49f76899j111212b02c14f192@mail.gmail.com> On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Thu Jan 7 15:05:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:05:23 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB, 07.01.2010 04:07: > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. > or PyErr_SetString? Certainly not safe. > Is there an easy way to find out? Release it and fix any crashes? Note that this isn't a safe solution, though, as some GIL requiring code may be platform specific. So a better approach might be to extract any obviously problematic stuff from the existing code (such as any exception handling, explicit ref-counting or object creation), and *then* try to release the GIL. Stefan From solipsis at pitrou.net Thu Jan 7 16:38:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 15:38:31 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or macros you want to call with the GIL released, and assess whether it is safe to call them. As already pointed out, no reference count should be changed, and generally no mutable container should be accessed, except if that container is known not to be referenced anywhere else (that would be the case for e.g. a list that your function has created and is busy populating). I agree that releasing the GIL when doing non-trivial regex searches is a worthwhile research, so please don't give up immediately :-) Regards Antoine Pitrou. From olemis at gmail.com Thu Jan 7 19:24:59 2010 From: olemis at gmail.com (Olemis Lang) Date: Thu, 7 Jan 2010 13:24:59 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> Message-ID: <24ea26601001071024m2abd988fuc77f94845d57483a@mail.gmail.com> On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am strongly -1 for modifying the classes in ?traditional? unittest > module [2]_ (except that I am strongly +1 for the package structure > WITHOUT TOUCHING anything else ...) , and the more I think about it I > am more convinced ... but anyway, this not a big deal (and in the end > what I think is not that relevant either ... :o). So ... > IOW, if I had all the freedom to implement it, after the pkg structure I'd do something like : {{{ #!python class TestResult: # Everything just the same def _is_relevant_tb_level(self, tb): return '__unittest' in tb.tb_frame.f_globals class BetterTestResult(TestResult): # Further code ... maybe ;o) # def _is_relevant_tb_level(self, tb): # This or anything else you might want to do ;o) # globs = tb.tb_frame.f_globals is_relevant = '__name__' in globs and \ globs["__name__"].startswith("unittest") del globs return is_relevant }}} that's what inheritance is for ;o) ... but quite probably that's not gonna happen, just a comment . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Ubuntu sustituye GIMP por F-Spot - http://feedproxy.google.com/~r/simelo-es/~3/-g48D6T6Ojs/ubuntu-sustituye-gimp-por-f-spot.html From martin at v.loewis.de Thu Jan 7 21:24:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:24:32 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: <4B464300.2020204@v.loewis.de> >> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question about releasing the > GIL during regexp searches? No, because the regex engine may also operate on buffers that start moving around when you release the GIL. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:27:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:27:09 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B46439D.9030604@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in place - if their contents changes, regex results may be confusing. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:31:21 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Message-ID: <4B464499.4020305@v.loewis.de> > I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process a file > called ast.txt. Where does this file come from ? Is it generated from > Python.asdl ? astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. HTH, Martin From foom at fuhm.net Thu Jan 7 21:36:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 7 Jan 2010 15:36:37 -0500 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B46439D.9030604@v.loewis.de> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> On Jan 7, 2010, at 3:27 PM, Martin v. L?wis wrote: >> I've been wondering whether it's possible to release the GIL in the >> regex engine during matching. > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Even if they stay in place - > if their contents changes, regex results may be confusing. It seems probably worthwhile to optimize for the common case of using the regexp engine on an immutable object of type "str" or "bytes", and allow releasing the GIL in *that* case, even if you have to keep it for the general case. James From martin at v.loewis.de Thu Jan 7 21:45:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:45:42 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> Message-ID: <4B4647F6.2060309@v.loewis.de> >>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even if they stay in place - >> if their contents changes, regex results may be confusing. > > It seems probably worthwhile to optimize for the common case of using > the regexp engine on an immutable object of type "str" or "bytes", and > allow releasing the GIL in *that* case, even if you have to keep it for > the general case. Right. This problem was the one that I thought of first. Thinking about these things is fairly difficult (to me, at least), so I think I could only tell whether I would consider a patch thread-safe that released the GIL around matching under selected circumstances - if I had the patch available. I don't see any obvious reason (assuming Guido's list of conditions holds - i.e. you are holding references to everything you access). Regards, Martin From ncoghlan at gmail.com Thu Jan 7 21:48:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Jan 2010 06:48:05 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DECB.7070907@agama.tv> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <4B45DECB.7070907@agama.tv> Message-ID: <4B464885.40806@gmail.com> Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a contributor agreement was essential for this particular contribution would depend on how much new code was needed for the backport, but the bulk of the copyright on the C RLock code would remain with Antoine regardless. However, sorting through the legalities of the contributor agreement really is the best way to make sure every is squared away nice and neatly from a legal point of view. After all, even if I was a lawyer (which I'm not, I'm just a developer with an interest in licensing issues), I still wouldn't be *your* lawyer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Jan 7 21:43:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 20:43:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the representation isn't moved away until the buffer is released. Regards Antoine. From martin at v.loewis.de Thu Jan 7 21:59:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:59:29 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B464B31.7040406@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin From johan.gill at agama.tv Thu Jan 7 14:16:59 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 14:16:59 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <4B45DECB.7070907@agama.tv> On 01/07/2010 01:23 PM, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. For corporate contributions, I believe the contributor > agreement needs to be signed by an authorised agent of the company - the > place to check that would probably be psf at python.org (that's the email > address for the PSF board). > > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) > > Regards, > Nick. > > Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Regards Johan Gill Agama Technologies From yoann.padioleau at facebook.com Thu Jan 7 22:07:55 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Thu, 7 Jan 2010 13:07:55 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464499.4020305@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> Message-ID: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> On Jan 7, 2010, at 12:31 PM, Martin v. L?wis wrote: >> I would like to use astgen.py to generate python classes corresponding to the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take as an argument a .asdl file, but apparently it instead process a file >> called ast.txt. Where does this file come from ? Is it generated from >> Python.asdl ? > > astgen.py is not used to process asdl files; ast.txt lives right next to > astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a reflection mechanism things that were generated from ast.txt, then there could be a mismatch. Where does ast.txt comes from ? Shouldn't it be generated itself from Python.adsl ? So we would have Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py containing all the UnarySub, Expression, classes that represents a Python AST. > > HTH, > Martin From martin at v.loewis.de Thu Jan 7 22:11:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 07 Jan 2010 22:11:36 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <4B464E08.5020703@v.loewis.de> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the representation isn't moved away until the buffer is > released. In 2.7, we currently get the buffer with bf_getreadbuffer. In 3.x, we have /* Release the buffer immediately --- possibly dangerous but doing something else would require some re-factoring */ PyBuffer_Release(&view); Even if we do use the new API, and correctly, it still might be confusing if the contents of the buffer changes underneath. Regards, Martin From martin at v.loewis.de Thu Jan 7 22:16:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 22:16:12 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> Message-ID: <4B464F1C.7020404@v.loewis.de> >> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose as a reflection mechanism things that were generated from > ast.txt, then there could be a mismatch. Where does ast.txt comes > from ? Shouldn't it be generated itself from Python.adsl ? What you may not be aware of is that Tools/compiler (and the compiler package that it builds on) are both unused and unmaintained. If the package stops working correctly - tough luck. > So we would have > > Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py > containing all the UnarySub, Expression, classes that represents a > Python AST. No - what actually happens in Python 3.x is this: both the compiler package and Tools/compiler are removed. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 01:10:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 01:10:35 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM Message-ID: <201001080110.35800.victor.stinner@haypocalc.com> Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading an UTF-8 text file including a BOM: #7185 (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with the UTF-8-SIG encoding, but it's possible to do better. I propose to improve open() (TextIOWrapper) by using the BOM to choose the right encoding. I think that only files opened in read only mode should support this new feature. *Read* the BOM in a *write* only file would cause unexpected behaviours. Since my proposition changes the result TextIOWrapper.read()/readline() for files starting with a BOM, we might introduce an option to open() to enable the new behaviour. But is it really needed to keep the backward compatibility? I wrote a proof of concept attached to the issue #7651. My patch only changes the behaviour of TextIOWrapper for reading files starting with a BOM. It doesn't work yet if a seek() is used before the first read. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Jan 8 01:52:20 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 16:52:20 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner wrote: > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". > > See recent issues related to reading an UTF-8 text file including a BOM: #7185 > (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with > the UTF-8-SIG encoding, but it's possible to do better. > > I propose to improve open() (TextIOWrapper) by using the BOM to choose the > right encoding. I think that only files opened in read only mode should > support this new feature. *Read* the BOM in a *write* only file would cause > unexpected behaviours. > > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? > > I wrote a proof of concept attached to the issue #7651. My patch only changes > the behaviour of TextIOWrapper for reading files starting with a BOM. It > doesn't work yet if a seek() is used before the first read. > > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Fri Jan 8 03:23:08 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 02:23:08 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46970C.9010306@mrabarnett.plus.com> Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > Alternatively, have a universal UTF-8/16/32 encoding, ie one that expects UTF-8, with or without BOM, or UTF-16/32 with BOM. > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> >> See recent issues related to reading an UTF-8 text file including a BOM: #7185 >> (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with >> the UTF-8-SIG encoding, but it's possible to do better. >> >> I propose to improve open() (TextIOWrapper) by using the BOM to choose the >> right encoding. I think that only files opened in read only mode should >> support this new feature. *Read* the BOM in a *write* only file would cause >> unexpected behaviours. >> >> Since my proposition changes the result TextIOWrapper.read()/readline() for >> files starting with a BOM, we might introduce an option to open() to enable >> the new behaviour. But is it really needed to keep the backward compatibility? >> >> I wrote a proof of concept attached to the issue #7651. My patch only changes >> the behaviour of TextIOWrapper for reading files starting with a BOM. It >> doesn't work yet if a seek() is used before the first read. >> From nick.bastin at gmail.com Fri Jan 8 04:11:03 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 7 Jan 2010 22:11:03 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> Message-ID: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs to do this on FreeBSD as well (adding bsd to the list of platforms for which this is performed "solves" the problem, but I don't pretend to know enough about either distutils or freebsd to determine if this is the correct solution). -- Nick From glyph at twistedmatrix.com Fri Jan 8 04:34:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 7 Jan 2010 22:34:36 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? It *is* crazy, but unfortunately rather common. Wikipedia has a good description of the issues: . Basically, some Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as being UTF-8, so it's become a convention to do that. That's not good enough, so you need to guess the encoding as well to make sure, but if there is a BOM and you can otherwise verify that the file is probably UTF-8 encoded, you should discard it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Jan 8 05:17:28 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 07 Jan 2010 23:17:28 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solaris and automatically adds '.' to the library_dirs, and > I suspect it just needs to do this on FreeBSD as well (adding bsd to > the list of platforms for which this is performed "solves" the > problem, but I don't pretend to know enough about either distutils or > freebsd to determine if this is the correct solution). I wouldn't say it needed discussion on the SIG: just create a bug report, with the tentative patch you have worked out, and get it assigned to Tarek. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGsdQACgkQ+gerLs4ltQ5BMQCgtV8snMXH/6dDwgdN4sIJljLd koYAoKq6c0tKsRSrITHcygu4Od9FVzF5 =BJaE -----END PGP SIGNATURE----- From guido at python.org Fri Jan 8 05:21:04 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 20:21:04 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > > file starting with a BOM, read()/readline() returns also the BOM whereas the > > BOM should be "ignored". > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > > It *is* crazy, but unfortunately rather common. ?Wikipedia has a good > description of the issues: > . ?Basically, some > Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as > being UTF-8, so it's become a convention to do that. ?That's not good > enough, so you need to guess the encoding as well to make sure, but if there > is a BOM and you can otherwise verify that the file is probably UTF-8 > encoded, you should discard it. That doesn't make sense. If the file isn't UTF-8 you can't see the BOM, because the BOM itself is UTF-8-encoded. (And yes, I know this happens. Doesn't mean we need to auto-guess by default; there are lots of issues e.g. what should happen after seeking to offset 0?) -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Fri Jan 8 07:06:16 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 08 Jan 2010 15:06:16 +0900 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from the user for each instance. But it should do something sane when reading such files. I can't really see any harm in throwing it away, especially since use of ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated IIRC. From tseaver at palladion.com Fri Jan 8 07:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 01:12:12 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> >> BOM should be "ignored". >> >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. > > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) The BOM should not be seekeable if the file is opened with the proposed "guess encoding from BOM" mode: it isn't properly part of the stream at all in that case. A UTF-8 BOM is an absurditiy, but it exists *everywhere* in the wild: Python would do wll to make it as easy as possible to consume such files, as well as the non-insane versions (UTF-16 / UTF-32 BOMs). In the best of all possible worlds, I would just try opening the file so: f = open('/path/to/file', 'r', encoding="DWIFM") and any BOM present would set the encoding for the remainder of the stream.. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGzLsACgkQ+gerLs4ltQ5+cwCdGfycPdj6+cPfD23vH644SpHL sI0AoLGD7nfgMEJdJhBr90yjQQHfDgcJ =js+2 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Fri Jan 8 08:55:27 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 8 Jan 2010 02:55:27 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. It's just that the UTF-8 decoding of the BOM at the start of a file should be "". > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I think it's pretty clear that the BOM should still be skipped in that case ... From martin at v.loewis.de Fri Jan 8 10:05:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:05:17 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F54D.9090103@v.loewis.de> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I think what Glyph meant is this: if a file starts with the UTF-8 signature, assume it's UTF-8. Then validate the assumption against the rest of the file also, and then process it as UTF-8. If the rest clearly is not UTF-8, assume that the UTF-8 signature is bogus. I understood this proposal as a general processing guideline, not something the io library should do (but, say, a text editor). FWIW, I'm personally in favor of using the UTF-8 signature. If people consider them crazy talk, that may be because UTF-8 can't possibly have a byte order - hence I call it a signature, not the BOM. As a signature, I don't consider it crazy at all. There is a long tradition of having magic bytes in files (executable files, Postscript, PDF, ... - see /etc/magic). Having a magic byte sequence for plain text to denote the encoding is useful and helps reducing moji-bake. This is the reason it's used on Windows: notepad would normally assume that text is in the ANSI code page, and for compatibility, it can't stop doing that. So the UTF-8 signature gives them an exit strategy. Regards, Martin From martin at v.loewis.de Fri Jan 8 10:06:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:06:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <201001080110.35800.victor.stinner@haypocalc.com> <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4B46F59A.5060905@v.loewis.de> > But it should do something sane when reading such files. I can't > really see any harm in throwing it away, especially since use of > ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated > IIRC. And indeed it does, when you open the file in the utf-8-sig encoding. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 10:08:30 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 10:08:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46970C.9010306@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46970C.9010306@mrabarnett.plus.com> Message-ID: <201001081008.30904.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 03:23:08, MRAB a ?crit : > Guido van Rossum wrote: > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > > talk. And for the other two, perhaps it would make more sense to have > > a separate encoding-guessing function that takes a binary stream and > > returns a text stream wrapping it with the proper encoding? > > Alternatively, have a universal UTF-8/16/32 encoding, ie one that > expects UTF-8, > with or without BOM, or UTF-16/32 with BOM. Do you mean open(filename, encoding="BOM")? I suppose that "BOM" would be a magical value specific to read a text file (open(filename, "r")), not a real codec? Otherwise which encoding should be used for open(filename, "w", encoding="BOM")? -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Fri Jan 8 10:10:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:10:23 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F67F.60604@v.loewis.de> > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". It depends. If you use the utf-8-sig encoding, it *will* ignore the UTF-8 signature. > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? Absolutely. And there is no need to produce a new option, but instead use the existing options: define an encoding that auto-detects the encoding from the family of BOMs. Maybe you call it encoding="sniff". Regards, Martin From martin at v.loewis.de Fri Jan 8 10:11:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:11:51 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: <4B46F6D7.1050301@v.loewis.de> Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. I'm fairly skeptical that anybody on distutils SIG is interested in details of the Python build process... Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 11:27:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:27:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081127.44044.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : (...) > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I wrote a new version of my patch (version 3): * don't change the default behaviour: use open(filename, encoding="BOM") to check the BOM is there is any * fix for seek(0): always ignore the BOM * add an unit test: check that the right encoding is detect, but also the the BOM is ignored (especially after a seek(0)) BOM encoding doesn't work for writing into a file, so open(filename, "w", encoding="BOM") raises a ValueError. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:31:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:31:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081131.37492.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 01:52:20, Guido van Rossum a ?crit : > And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? I choosed to modify open()+TextIOWrapper instead of writing a new function because I would like to avoid an extra read operation (syscall) on the file. With my implementation, no specific read operation is needed to detect the BOM. The BOM is simply checked in the first _read_chunk(). -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:40:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:40:28 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Improve_open=28=29_to_support_reading?= =?iso-8859-1?q?_file_starting_with=09an_unicode_BOM?= In-Reply-To: <4B46F67F.60604@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> Message-ID: <201001081140.28124.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : > > Builtin open() function is unable to open an UTF-16/32 file starting with > > a BOM if the encoding is not specified (raise an unicode error). For an > > UTF-8 file starting with a BOM, read()/readline() returns also the BOM > > whereas the BOM should be "ignored". > > It depends. If you use the utf-8-sig encoding, it *will* ignore the > UTF-8 signature. Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to remove the BOM after the first read (much harder if you use a module like ConfigParser or csv). > > Since my proposition changes the result TextIOWrapper.read()/readline() > > for files starting with a BOM, we might introduce an option to open() to > > enable the new behaviour. But is it really needed to keep the backward > > compatibility? > > Absolutely. And there is no need to produce a new option, but instead > use the existing options: define an encoding that auto-detects the > encoding from the family of BOMs. Maybe you call it encoding="sniff". Good idea, I choosed open(filename, encoding="BOM"). -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri Jan 8 15:27:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:27:42 +0000 (UTC) Subject: [Python-Dev] GIL required for _all_ Python calls? References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: > > Even if we do use the new API, and correctly, it still might be > confusing if the contents of the buffer changes underneath. Well, no more confusing than when you compute a SHA1 hash or zlib- compress the buffer, is it? Regards Antoine From solipsis at pitrou.net Fri Jan 8 15:34:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:34:26 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any Well, I think if we implement this the default behaviour *should* be changed. It looks a bit senseless to have two different "auto-choose" options, one with encoding=None and one with encoding="BOM". Regards Antoine. From guido at python.org Fri Jan 8 16:48:44 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:48:44 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 6:27 AM, Antoine Pitrou wrote: > Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: >> >> Even if we do use the new API, and correctly, it still might be >> confusing if the contents of the buffer changes underneath. > > Well, no more confusing than when you compute a SHA1 hash or zlib- > compress the buffer, is it? That depends. Algorithms that make exactly one pass over the buffer will run fine (maybe producing a meaningless result). But the regex matcher may scan the buffer repeatedly (for backtracking purposes) and it would take a considerable analysis to prove that cannot mess up its internal data structures if the data underneath changes. (I give it a decent chance that it's fine, but since it was written without ever considering this possibility I'm not 100% sure.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:52:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:52:48 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 11:55 PM, Glyph Lefkowitz wrote: > I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. And I'm saying that it is, with as much certainty as we can ever guess the encoding of a file. > If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. ?It's just that the UTF-8 decoding of the BOM at the start of a file should be "". Sure, a Latin-1-encoded file could start with the same pattern that is a UTF-8-encoded BOM. But at that point, a UTF-16-encoded file is also valid Latin-1. The question was in the context of encoding-guessing; if we're guessing, a UTF-8-encoded BOM cannot signify anything else but UTF-8. (Ditto for UTF-16 and UTF-32 BOMs.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:54:15 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:54:15 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> >> I wrote a new version of my patch (version 3): >> >> ?* don't change the default behaviour: use open(filename, encoding="BOM") to >> check the BOM is there is any > > Well, I think if we implement this the default behaviour *should* be changed. > It looks a bit senseless to have two different "auto-choose" options, one with > encoding=None and one with encoding="BOM". Well there *are* two different auto options: use the environment variables (LANG etc.) or inspect the contents of the file. I think it would be useful to have ways to specify both. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:56:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:56:46 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 1:05 AM, "Martin v. L?wis" wrote: >>> It *is* crazy, but unfortunately rather common. ?Wikipedia has a good >>> description of the issues: >>> . ?Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. ?That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. > > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Sure. I said "crazy talk" only to stir up discussion. Which worked. :-) Also, I don't want Python's default behavior to change -- sniffing the encoding should be a separate option. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:59:45 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:59:45 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: > The BOM should not be seekeable if the file is opened with the proposed > "guess encoding from BOM" mode: ?it isn't properly part of the stream at > all in that case. This feels about right to me. There are still questions though: immediately after opening a file with a BOM, what should .tell() return? And regardless of that, .seek(0) should put the file in that same initial state. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Jan 8 17:03:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:03:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > > Well, I think if we implement this the default behaviour *should* be changed. > > It looks a bit senseless to have two different "auto-choose" options, one with > > encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Yes, perhaps. In the context of open() however I think it would be helpful to change the default. Moreover, reading the BOM is certainly much more reliable than our current guessing based on the locale or the "device encoding". Regards Antoine. From mal at egenix.com Fri Jan 8 17:25:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 17:25:22 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B475C72.1010207@egenix.com> Guido van Rossum wrote: > On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: >> Victor Stinner haypocalc.com> writes: >>> >>> I wrote a new version of my patch (version 3): >>> >>> * don't change the default behaviour: use open(filename, encoding="BOM") to >>> check the BOM is there is any >> >> Well, I think if we implement this the default behaviour *should* be changed. >> It looks a bit senseless to have two different "auto-choose" options, one with >> encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Shouldn't this encoding guessing be a separate function that you call on either a file or a seekable stream ? After all, detecting encodings is just as useful to have for non-file streams. You'd then avoid having to stuff everything into a single function call and also open up the door for more complex application specific guess work or defaults. The whole process would then have two steps: 1. guess encoding import codecs encoding = codecs.guess_file_encoding(filename) 2. open the file with the found encoding f = open(filename, encoding=encoding) For seekable streams f, you'd have: 1. guess encoding import codecs encoding = codecs.guess_stream_encoding(f) 2. wrap the stream with a reader for the found encoding reader_class = codecs.getreader(encoding) g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Jan 8 17:31:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:31:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver palladion.com> wrote: > > The BOM should not be seekeable if the file is opened with the proposed > > "guess encoding from BOM" mode: it isn't properly part of the stream at > > all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? tell() in the context of text I/O is specified to return an "opaque cookie". So whatever value it returns would probably be fine, as long as seeking to that value leaves the file in an acceptable state. Rewinding (seeking to 0) in the presence of a BOM is already reasonably supported by the TextIOWrapper object: >>> dec = codecs.getincrementaldecoder('utf-16')() >>> dec.decode(b'\xff\xfea\x00b\x00') 'ab' >>> dec.decode(b'\xff\xfea\x00b\x00') '\ufeffab' >>> >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' There are tests for this in test_io.py (test_encoded_writes, line 1929, and test_append_bom and test_seek_bom, line 2045). Regards Antoine. From python at mrabarnett.plus.com Fri Jan 8 17:47:18 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 16:47:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081127.44044.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B476196.4080409@mrabarnett.plus.com> Victor Stinner wrote: > Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : > (...) >> (And yes, I know this happens. Doesn't mean we need to auto-guess by >> default; there are lots of issues e.g. what should happen after >> seeking to offset 0?) > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any > * fix for seek(0): always ignore the BOM > * add an unit test: check that the right encoding is detect, but also the the > BOM is ignored (especially after a seek(0)) > > BOM encoding doesn't work for writing into a file, so open(filename, "w", > encoding="BOM") raises a ValueError. > I think it's similar to universal newline mode. You can tell it that you're reading UTF-something-encoded text (common forms only). The preference is UTF-8, but it could be UTF-8-sig (from Windows), or possibly UTF-16/32, which really need a BOM because there are multiple bytes per codepoint, so the bytes could be big-endian or little-endian. The BOM (or signature) tells you what the encoding is, defaulting to UTF-8 if there's none. If it subsequently raises a DecodeError, then so be it! Maybe there should also be a way of determining what encoding it decided it was, so that you can then write a new file in that same encoding. From status at bugs.python.org Fri Jan 8 18:07:13 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Jan 2010 18:07:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100108170714.014B078669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/01/10 - 01/08/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2544 open (+27) / 16937 closed (+15) / 19481 total (+42) Open issues with patches: 1017 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2509 (+27) pending 34 ( +0) Issues Created Or Reopened (43) _______________________________ Extended slicing with classic class behaves strangely 01/07/10 http://bugs.python.org/issue7532 reopened mark.dickinson patch optparse library documentation has an insignificant formatting i 01/01/10 CLOSED http://bugs.python.org/issue7618 created vazovsky patch imaplib shouldn't use cause DeprecationWarnings in 2.6 01/01/10 CLOSED http://bugs.python.org/issue7619 created djc Vim syntax highlight 01/02/10 http://bugs.python.org/issue7620 created july patch Test issue 01/02/10 CLOSED http://bugs.python.org/issue7621 created georg.brandl [patch] improve unicode methods: split() rsplit() and replace() 01/03/10 http://bugs.python.org/issue7622 created flox patch PropertyType missing in Lib/types.py 01/03/10 CLOSED http://bugs.python.org/issue7623 created wplappert isinstance(... ,collections.Callable) fails with oldstyle class 01/03/10 http://bugs.python.org/issue7624 created rgammans bytearray needs more tests for "b.some_method()[0] is not b" 01/03/10 http://bugs.python.org/issue7625 created flox patch Entity references without semicolon in HTMLParser 01/03/10 CLOSED http://bugs.python.org/issue7626 created stefan.schweizer mailbox.MH.remove() lock handling is broken 01/04/10 http://bugs.python.org/issue7627 created sraustein round() doesn't work correctly in 3.1.1 01/04/10 CLOSED http://bugs.python.org/issue7628 created bkovt Compiling with mingw32 gcc, content of variable "extra_postargs" 01/04/10 CLOSED http://bugs.python.org/issue7629 created popelkopp Strange behaviour of decimal.Decimal 01/04/10 CLOSED http://bugs.python.org/issue7630 created parmax undefined label: bltin-file-objects 01/04/10 CLOSED http://bugs.python.org/issue7631 created ezio.melotti dtoa.c: oversize b in quorem 01/04/10 http://bugs.python.org/issue7632 created skrah decimal.py: type conversion in context methods 01/04/10 http://bugs.python.org/issue7633 created skrah patch, easy next/previous links in documentation skip some sections 01/05/10 CLOSED http://bugs.python.org/issue7634 created gagenellina 19.6 xml.dom.pulldom doc: stub? 01/05/10 http://bugs.python.org/issue7635 created tjreedy Add a set update action to optparse 01/05/10 CLOSED http://bugs.python.org/issue7636 created hardkrash patch Improve 19.5. xml.dom.minidom doc 01/05/10 http://bugs.python.org/issue7637 created tjreedy Counterintuitive str.splitlines() inconsistency. 01/05/10 CLOSED http://bugs.python.org/issue7638 created vencabot_teppoo bdist_msi fails on files with long names 01/05/10 http://bugs.python.org/issue7639 created mmm77 buffered io seek() buggy 01/05/10 http://bugs.python.org/issue7640 created pakal patch Built-in Formatter accepts undocumented presentation type 01/06/10 CLOSED http://bugs.python.org/issue7641 created lrekucki [patch] Minor improvement in os.system doc 01/06/10 http://bugs.python.org/issue7642 created Val patch What is an ASCII linebreak? 01/06/10 http://bugs.python.org/issue7643 created flox bug in nntplib.body() method with possible fix 01/06/10 http://bugs.python.org/issue7644 created mdmullins easy test_distutils fails on Windows XP 01/06/10 http://bugs.python.org/issue7645 created austin987 test_telnetlib fails in Windows XP 01/06/10 http://bugs.python.org/issue7646 created austin987 Add statvfs flags to the posix module 01/06/10 http://bugs.python.org/issue7647 created ajax at redhat.com patch test_urllib2 fails on Windows if not run from C: 01/06/10 http://bugs.python.org/issue7648 created austin987 easy "u'%c' % char" broken for chars in range '\x80'-'\xFF' 01/07/10 http://bugs.python.org/issue7649 created ezio.melotti patch, easy test_uuid is invalid 01/07/10 http://bugs.python.org/issue7650 created austin987 Python3: guess text file charset using the BOM 01/07/10 http://bugs.python.org/issue7651 created haypo patch Merge C version of decimal into py3k. 01/07/10 http://bugs.python.org/issue7652 created mark.dickinson Fix description of the way the PythonPath Windows registry key w 01/07/10 CLOSED http://bugs.python.org/issue7653 created BoarGules patch, needs review [patch] Enable additional bytes and memoryview tests. 01/07/10 http://bugs.python.org/issue7654 created flox patch PEP 374 Title usage & script redirection typo fixes 01/07/10 CLOSED http://bugs.python.org/issue7655 created bab9e9 patch test_hashlib fails on some installations (specifically Neal's re 01/08/10 http://bugs.python.org/issue7656 created r.david.murray test_ctypes failure on AIX 5.3 01/08/10 http://bugs.python.org/issue7657 created mallayya patch OS X pythonw.c compile error with 10.4 or earlier deployment tar 01/08/10 http://bugs.python.org/issue7658 created ned.deily Problems with attribute assignment on object instances 01/08/10 http://bugs.python.org/issue7659 created pakal Issues Now Closed (40) ______________________ shutil fails when copying to NTFS in Linux 762 days http://bugs.python.org/issue1545 benjamin.peterson patch Test 751 days http://bugs.python.org/issue1619 georg.brandl Windows Registry issue with 2.5.2 AMD64 msi 645 days http://bugs.python.org/issue2539 loewis Extension module build fails for MinGW: missing vcvarsall.bat 618 days http://bugs.python.org/issue2698 Daniel26 optparse print_usage(),.. methods are not documented 542 days http://bugs.python.org/issue3340 ezio.melotti reference leaks in test_distutils 500 days http://bugs.python.org/issue3660 ezio.melotti patch _sha256 et al. encode to UTF-8 by default 20 days http://bugs.python.org/issue3745 lemburg 26backport Add Option to Bind to a Local IP Address in httplib.py 464 days http://bugs.python.org/issue3972 gregory.p.smith patch, easy no reference for optparse methods 388 days http://bugs.python.org/issue4635 ezio.melotti PyArg_Parse* should raise TypeError for float parsed with intege 339 days http://bugs.python.org/issue5080 mark.dickinson patch Don't use PyLong_SHIFT with _PyLong_AsScaledDouble() 282 days http://bugs.python.org/issue5576 mark.dickinson patch PyFrame_GetLineNumber 244 days http://bugs.python.org/issue5954 georg.brandl patch PyCode_NewEmpty 244 days http://bugs.python.org/issue5959 georg.brandl patch Add non-command help topics to help completion of cmd.Cmd 241 days http://bugs.python.org/issue5991 georg.brandl patch make error 231 days http://bugs.python.org/issue6067 georg.brandl no longer possible to hash arrays 229 days http://bugs.python.org/issue6071 benjamin.peterson patch zipfile: Invalid argument when opening zero-sized files 173 days http://bugs.python.org/issue6511 r.david.murray use different mechanism for pythonw on osx 127 days http://bugs.python.org/issue6834 ned.deily easy Py3k doc: "from __future__ import division" not necessary 32 days http://bugs.python.org/issue7432 ezio.melotti cPickle: stack underflow in load_pop() 31 days http://bugs.python.org/issue7455 pitrou patch crash in str.rfind() with an invalid start value 25 days http://bugs.python.org/issue7458 pitrou patch Implement fastsearch algorithm for rfind/rindex 26 days http://bugs.python.org/issue7462 ezio.melotti patch GZipFile.readline too slow 20 days http://bugs.python.org/issue7471 pitrou patch ssl module documentation: SSLSocket.unwrap description shown twi 5 days http://bugs.python.org/issue7592 georg.brandl Installation - 2 tests failed: test_cmd_line test_xmlrpc 7 days http://bugs.python.org/issue7601 ezio.melotti optparse library documentation has an insignificant formatting i 2 days http://bugs.python.org/issue7618 ezio.melotti patch imaplib shouldn't use cause DeprecationWarnings in 2.6 1 days http://bugs.python.org/issue7619 djc Test issue 0 days http://bugs.python.org/issue7621 ezio.melotti PropertyType missing in Lib/types.py 0 days http://bugs.python.org/issue7623 benjamin.peterson Entity references without semicolon in HTMLParser 2 days http://bugs.python.org/issue7626 r.david.murray round() doesn't work correctly in 3.1.1 0 days http://bugs.python.org/issue7628 benjamin.peterson Compiling with mingw32 gcc, content of variable "extra_postargs" 0 days http://bugs.python.org/issue7629 r.david.murray Strange behaviour of decimal.Decimal 0 days http://bugs.python.org/issue7630 mark.dickinson undefined label: bltin-file-objects 0 days http://bugs.python.org/issue7631 pitrou next/previous links in documentation skip some sections 0 days http://bugs.python.org/issue7634 georg.brandl Add a set update action to optparse 3 days http://bugs.python.org/issue7636 r.david.murray patch Counterintuitive str.splitlines() inconsistency. 0 days http://bugs.python.org/issue7638 flox Built-in Formatter accepts undocumented presentation type 0 days http://bugs.python.org/issue7641 eric.smith Fix description of the way the PythonPath Windows registry key w 0 days http://bugs.python.org/issue7653 r.david.murray patch, needs review PEP 374 Title usage & script redirection typo fixes 0 days http://bugs.python.org/issue7655 georg.brandl patch Top Issues Most Discussed (10) ______________________________ 22 [patch] improve unicode methods: split() rsplit() and replace() 5 days open http://bugs.python.org/issue7622 14 Test suite emits many DeprecationWarnings when -3 is enabled 91 days open http://bugs.python.org/issue7092 13 unicode_escape codec does not escape quotes 8 days open http://bugs.python.org/issue7615 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 0 days open http://bugs.python.org/issue7658 8 Merge C version of decimal into py3k. 1 days open http://bugs.python.org/issue7652 8 "u'%c' % char" broken for chars in range '\x80'-'\xFF' 2 days open http://bugs.python.org/issue7649 8 decimal.py: type conversion in context methods 4 days open http://bugs.python.org/issue7633 8 Patch for [ 735515 ] urllib2 should cache 301 redir 906 days open http://bugs.python.org/issue1755841 7 Test issue 0 days closed http://bugs.python.org/issue7621 7 Cannot use both read and readline method in same ZipExtFile obj 8 days open http://bugs.python.org/issue7610 From tseaver at palladion.com Fri Jan 8 22:09:54 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:09:54 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: >> The BOM should not be seekeable if the file is opened with the proposed >> "guess encoding from BOM" mode: it isn't properly part of the stream at >> all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? And regardless of that, .seek(0) should put the file in that > same initial state. I think the behavior should be something like: >>> f = open('/path/to/maybe-BOM-encoded-file', 'r', encoding='BOM') >>> f.tell() 0L >>> f.seek(-1) >>> f.tell() # count of unicode chars in decoded stream 45L >>> f.seek(0) >>> f.read(1) # read first unicode char decoded from stream. 'A' In other words, the BOM is not readable / seekable at all: it is invisible to the consumer of the decoded stream. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHnyIACgkQ+gerLs4ltQ6s3QCgznD+7FbUzfCbe5TS6OcoXjMg rdgAoJAMEXe2xwLCIwJaZ6XA6rVyTIAi =oXb3 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:19:10 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:19:10 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B475C72.1010207@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: > Shouldn't this encoding guessing be a separate function that you call > on either a file or a seekable stream ? > > After all, detecting encodings is just as useful to have for non-file > streams. Other stream sources typically have out-of-band ways to signal the encoding: only when reading from the filesystem do we pretty much *have* to guess, and in that case the BOM / signature is the best heuristic we have. Also, some non-file streams are not seekable, and so can't be guessed via a pre-pass. > You'd then avoid having to stuff everything into > a single function call and also open up the door for more complex > application specific guess work or defaults. > > The whole process would then have two steps: > > 1. guess encoding > > import codecs > encoding = codecs.guess_file_encoding(filename) Filename is not enough information: or do you mean that API to actually open the stream? > 2. open the file with the found encoding > > f = open(filename, encoding=encoding) > > For seekable streams f, you'd have: > > 1. guess encoding > > import codecs > encoding = codecs.guess_stream_encoding(f) > > 2. wrap the stream with a reader for the found encoding > > reader_class = codecs.getreader(encoding) > g = reader_class(f) > Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoU4ACgkQ+gerLs4ltQ5o3QCeLOJ7J91E+5f66vhgu1BUhYh4 9UgAnR2IeCd0BCsPez8ZilGNHJfhRn3Y =SoPb -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:14:59 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:14:59 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> It *is* crazy, but unfortunately rather common. Wikipedia has a good >>> description of the issues: >>> . Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. If the programmer opens the file using a "guess using the BOM" encoding, Python should *not* attempt to verify that the file is properly encoded: it should check for (and consume) any BOM, and then return a stream which uses the encoding inferred from the BOM. Any errors should be handled later, when characters are read, exactly as if the file had been opened with the same encoding guessed from the BOM. > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Agreed. Having that marker at the start of the file makes interop with other tools *much* easier. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoFMACgkQ+gerLs4ltQ73dACffwUfyh6Q9vUnKYf367QFjNcU RRMAoNuKCWEx7j+MSdTv+UjhAPynBc14 =uAX6 -----END PGP SIGNATURE----- From eric at trueblade.com Fri Jan 8 22:40:47 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 8 Jan 2010 16:40:47 -0500 (EST) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? >> >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. But what if the file were in (for example) a zip file? I think you definitely want to have access to this functionality outside of open(). Eric. From foom at fuhm.net Fri Jan 8 22:49:23 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 8 Jan 2010 16:49:23 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Jan 8, 2010, at 4:14 PM, Tres Seaver wrote: >> I understood this proposal as a general processing guideline, not >> something the io library should do (but, say, a text editor). >> >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly >> have >> a byte order - hence I call it a signature, not the BOM. As a >> signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote >> the >> encoding is useful and helps reducing moji-bake. This is the reason >> it's >> used on Windows: notepad would normally assume that text is in the >> ANSI >> code page, and for compatibility, it can't stop doing that. So the >> UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop > with > other tools *much* easier. Putting the BOM at the beginning of UTF-8 text files is not a good idea, it makes interop much *worse* on a unix system, not better. Without the BOM, most commands do the right thing with UTF-8 text. E.g. to concatenate two files: $ cat file-1 file-2 > file-3 With a BOM at the beginning of the file, it won't work right. Of course, you could modify "cat" (and every other stream processing command) to know how to consume and emit BOMs, and omit the extra one that would show up in the middle of the stream...but even that can't work; what about: $ (cat file-1; cat file-2) > file-3. Should the shell now know that when you run multiple commands, it should eat the BOM emitted from the second command? Basically, using a BOM in a utf-8 file is just not a good idea: it completely ruins interop with every standard unix tool. This is not to say that Python shouldn't have a way to read a file with a UTF-8 BOM: it just shouldn't encourage you to *write* such files. James From mal at egenix.com Fri Jan 8 22:51:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 22:51:26 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <4B47A8DE.1080401@egenix.com> Tres Seaver wrote: > M.-A. Lemburg wrote: > >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? > >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. Sure there are non-seekable file streams, but at least when using StringIO-type streams you don't have that restriction. An encoding detection function would provide more use in other cases as well, so instead of hiding away the functionality in the open() constructor, I'm suggesting to make expose it via the codecs module. Applications can then use it where necessary and also provide their own defaults, using other heuristics. >> You'd then avoid having to stuff everything into >> a single function call and also open up the door for more complex >> application specific guess work or defaults. > >> The whole process would then have two steps: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_file_encoding(filename) > > Filename is not enough information: or do you mean that API to actually > open the stream? Yes. The API would open the file, guess the encoding and then close it again. If you don't want that to happen, you could use the second API I mentioned below on the already open file. Note that this function could detect a lot more encodings with reasonably high probability than just BOM-prefixed ones, e.g. we could also add support to detect encoding declarations such as the ones we use in Python source files. >> 2. open the file with the found encoding > >> f = open(filename, encoding=encoding) > >> For seekable streams f, you'd have: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_stream_encoding(f) I forgot to mention: This API needs to position the file pointer to the start of the first data byte. >> 2. wrap the stream with a reader for the found encoding > >> reader_class = codecs.getreader(encoding) >> g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Fri Jan 8 22:59:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:59:04 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote: >>> Shouldn't this encoding guessing be a separate function that you call >>> on either a file or a seekable stream ? >>> >>> After all, detecting encodings is just as useful to have for non-file >>> streams. >> Other stream sources typically have out-of-band ways to signal the >> encoding: only when reading from the filesystem do we pretty much >> *have* to guess, and in that case the BOM / signature is the best >> heuristic we have. Also, some non-file streams are not seekable, and so >> can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). If the application expects a possibly-BOM-signature-marked file, but you pass it mismatched garbage: >>> f = open('some.zip', encoding='BOM") the error handling should be the same as if you passed any other mismatched encoding: >>> f = open('some.zip', encoding='UTF8') i.e., you discover the error when you try to read from the (non)encoded stream, not when you open it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHqpwACgkQ+gerLs4ltQ7uAACeKEc+WT4TASGcVl1Hfqe6L9La I6EAn1pJtngtLWPdothGbYB+zUabEvTW =TjBK -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Fri Jan 8 23:10:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 23:10:32 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: <201001082310.33029.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 22:40:47, Eric Smith a ?crit : > >> Shouldn't this encoding guessing be a separate function that you call > >> on either a file or a seekable stream ? > >> > >> After all, detecting encodings is just as useful to have for non-file > >> streams. > > > > Other stream sources typically have out-of-band ways to signal the > > encoding: only when reading from the filesystem do we pretty much > > *have* to guess, and in that case the BOM / signature is the best > > heuristic we have. Also, some non-file streams are not seekable, and so > > can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). FYI my patch (encoding="BOM") is implemented in TextIOWrapper, and TextIOWrapper takes a binary stream as input, not a filename. -- Victor Stinner http://www.haypocalc.com/ From yoann.padioleau at facebook.com Fri Jan 8 23:59:52 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Fri, 8 Jan 2010 14:59:52 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464F1C.7020404@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: On Jan 7, 2010, at 1:16 PM, Martin v. L?wis wrote: >>> astgen.py is not used to process asdl files; ast.txt lives right >>> next to astgen.py. Instead, the asdl file is processed by >>> Parser/asdl_c.py. >> >> Yes, I know that. That's why I asked about the relation between >> ast.txt and Python.adsl. If internally the parser uses the .adsl, but >> expose as a reflection mechanism things that were generated from >> ast.txt, then there could be a mismatch. Where does ast.txt comes >> from ? Shouldn't it be generated itself from Python.adsl ? > > What you may not be aware of is that Tools/compiler (and the > compiler package that it builds on) are both unused and unmaintained. I see. So if people want to analyze python code they have to use other tools (like rope?) ? or use reflection ? > > If the package stops working correctly - tough luck. > >> So we would have >> >> Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py >> containing all the UnarySub, Expression, classes that represents a >> Python AST. > > No - what actually happens in Python 3.x is this: both the compiler > package and Tools/compiler are removed. Ok. I will then create my own ast classes generator. Thanks. > > Regards, > Martin From g.brandl at gmx.net Sat Jan 9 00:10:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 09 Jan 2010 00:10:24 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: Am 08.01.2010 22:14, schrieb Tres Seaver: >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly have >> a byte order - hence I call it a signature, not the BOM. As a signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote the >> encoding is useful and helps reducing moji-bake. This is the reason it's >> used on Windows: notepad would normally assume that text is in the ANSI >> code page, and for compatibility, it can't stop doing that. So the UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop with > other tools *much* easier. Except if only 50% of the other tools support the signature. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jan 9 00:57:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 00:57:46 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: <4B47C67A.3020302@v.loewis.de> > I see. So if people want to analyze python code they have to use > other tools (like rope?) ? or use reflection ? Correct. One such tool might be the true Python compiler, along with the _ast module. Regards, Martin From victor.stinner at haypocalc.com Sat Jan 9 00:59:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 00:59:00 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM Message-ID: <201001090059.00848.victor.stinner@haypocalc.com> Hi, Thanks for all the answers! I will try to sum up all ideas here. (1) Change default open() behaviour or make it optional? Guido would like to add an option and keep open() unchanged. He wrote that checking for BOM and using system locale are too much different to be the same option (encoding=None). Antoine would like to check BOM by default, because both options (system locale vs checking for BOM) is the same thing. About Antoine choice (encoding=None): which file modes would check for a BOM? I would like to answer only the read only mode, but then open(filename, "r") and open(filename, "r+") would behave differently? => 1 point for Guido (encoding="BOM" is more explicit) Antoine choice has the advantage of directly support UTF-8+BOM, UTF-16 and UTF-32 for all applications and all modules using open(filename). => 1 point for Antoine (2) Check for a BOM while reading or detect it before? Everybody agree that checking BOM is an interesting option and should not be limited to open(). Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file name or a binary file-like object: it returns the encoding and seek to the file start or just after the BOM. I dislike this function because it requires extra file operations (open (optional), read() and seek()) and it doesn't work if the file is not seekable (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to avoid extra file operations. Note: I implemented the BOM check in TextIOWrapper; so it's already usable for any file-like object. (3) tell() and seek() on a text file starting with a BOM To be consistent with Antoine example: >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' TextIOWrapper: * tell() should return zero at file start, * seek(0) should go be to file start, * and the BOM should always be "ignored". I mean: with open("utf8bom.txt", encoding="BOM") as fp: assert fp.tell() == 0 text = fp.read() # no BOM here fp.seek(0) assert fp.read() == text -- About my patch: - BOM check is explicit: open(filebame, encoding="BOM") - tell() / seek(0) works as expected - BOM bytes are always skipped in read() / readlines() result Hum, I don't know if this email can be called a sum up ;-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 01:09:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 00:09:48 +0000 (UTC) Subject: [Python-Dev] Quick sum up about open() + BOM References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: Hello Victor, Victor Stinner haypocalc.com> writes: > > (1) Change default open() behaviour or make it optional? > [...] > > Antoine would like to check BOM by default, because both options (system > locale vs checking for BOM) is the same thing. To be clear, I am not saying it is the same thing. What I think is that it would be a mistake to use a mildly unreliable heuristic by default (the locale + device encoding heuristic) but refuse to trust a more reliable heuristic (the BOM-based detection algorithm). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Jan 9 01:14:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 00:14:39 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47CA6F.5000607@voidspace.org.uk> On 09/01/2010 00:09, Antoine Pitrou wrote: > Hello Victor, > > Victor Stinner haypocalc.com> writes: > >> (1) Change default open() behaviour or make it optional? >> >> > [...] > >> Antoine would like to check BOM by default, because both options (system >> locale vs checking for BOM) is the same thing. >> > To be clear, I am not saying it is the same thing. What I think is that it would > be a mistake to use a mildly unreliable heuristic by default (the locale + > device encoding heuristic) but refuse to trust a more reliable heuristic (the > BOM-based detection algorithm). > I concur. On Windows both UTF-8 and signature are very common, yet the platform default is the truly awful CP1252. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From floris.bruynooghe at gmail.com Sat Jan 9 01:26:05 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 9 Jan 2010 00:26:05 +0000 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B46F6D7.1050301@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> <4B46F6D7.1050301@v.loewis.de> Message-ID: <20100109002605.GB13536@laurie.devork> On Fri, Jan 08, 2010 at 10:11:51AM +0100, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > I think this problem probably needs to move over to distutils-sig, as > > it doesn't seem to be specific to the way that Python itself uses > > distutils. > > I'm fairly skeptical that anybody on distutils SIG is interested in > details of the Python build process... Uh, hum. Unfounded skepticism. ;-) But as said filing a bug sounds better in this case. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From v+python at g.nevcal.com Sat Jan 9 01:47:38 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 16:47:38 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47D22A.8070305@g.nevcal.com> On approximately 1/8/2010 3:59 PM, came the following characters from the keyboard of Victor Stinner: > Hi, > > Thanks for all the answers! I will try to sum up all ideas here. One concern I have with this implementation encoding="BOM" is that if there is no BOM it assumes UTF-8. That is probably a good assumption in some circumstances, but not in others. * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE encoded files include a BOM. It is only required that UTF-16 and UTF-32 (cases where the endianness is unspecified) contain a BOM. Hence, it might be that someone would expect a UTF-16LE (or any of the formats that don't require a BOM, rather than UTF-8), but be willing to accept any BOM-discriminated format. * Potentially, this could be expanded beyond the various Unicode encodings... one could envision that a program whose data files historically were in any particular national language locale, could want to be enhance to accept Unicode, and could declare that they will accept any BOM-discriminated format, but want to default, in the absence of a BOM, to the original national language locale that they historically accepted. That would provide a migration path for their old data files. So the point is, that it might be nice to have "BOM-otherEncodingForDefault" for each other encoding that Python supports. Not sure that is the right API, but I think it is expressive enough to handle the cases above. Whether the cases solve actual problems or not, I couldn't say, but they seem like reasonable cases. It would, of course, be nicest if OS metadata had been invented way back when, for all OSes, such that all text files were flagged with their encoding... then languages could just read the encoding and do the right thing! But we live in the real world, instead. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From python at mrabarnett.plus.com Sat Jan 9 02:12:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 09 Jan 2010 01:12:28 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B47D7FC.4070703@mrabarnett.plus.com> Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. > > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. > What about listing the possible encodings? It would try each in turn until it found one where the BOM matched or had no BOM: my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') or is that taking it too far? From martin at v.loewis.de Sat Jan 9 02:23:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 02:23:07 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47CA6F.5000607@voidspace.org.uk> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> Message-ID: <4B47DA7B.6050505@v.loewis.de> >>> Antoine would like to check BOM by default, because both options >>> (system locale vs checking for BOM) is the same thing. >>> >> To be clear, I am not saying it is the same thing. What I think is >> that it would be a mistake to use a mildly unreliable heuristic by >> default (the locale + device encoding heuristic) but refuse to >> trust a more reliable heuristic (the BOM-based detection >> algorithm). >> > > I concur. On Windows both UTF-8 and signature are very common, yet > the platform default is the truly awful CP1252. While I would support combining BOM detection in the case where a file is opened for reading and no encoding is specified, I see two problems: a) if a seek operations is performed before having looked at the BOM, no determination would have been made b) what encoding should it use on writing? Regards, Martin From v+python at g.nevcal.com Sat Jan 9 02:49:12 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 17:49:12 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <4B47E098.6030703@g.nevcal.com> On approximately 1/8/2010 5:12 PM, came the following characters from the keyboard of MRAB: > Glenn Linderman wrote: >> On approximately 1/8/2010 3:59 PM, came the following characters from >> the keyboard of Victor Stinner: >>> Hi, >>> >>> Thanks for all the answers! I will try to sum up all ideas here. >> >> One concern I have with this implementation encoding="BOM" is that if >> there is no BOM it assumes UTF-8. That is probably a good assumption >> in some circumstances, but not in others. >> >> * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE >> encoded files include a BOM. It is only required that UTF-16 and >> UTF-32 (cases where the endianness is unspecified) contain a BOM. >> Hence, it might be that someone would expect a UTF-16LE (or any of >> the formats that don't require a BOM, rather than UTF-8), but be >> willing to accept any BOM-discriminated format. >> >> * Potentially, this could be expanded beyond the various Unicode >> encodings... one could envision that a program whose data files >> historically were in any particular national language locale, could >> want to be enhance to accept Unicode, and could declare that they >> will accept any BOM-discriminated format, but want to default, in the >> absence of a BOM, to the original national language locale that they >> historically accepted. That would provide a migration path for their >> old data files. >> >> So the point is, that it might be nice to have >> "BOM-otherEncodingForDefault" for each other encoding that Python >> supports. Not sure that is the right API, but I think it is >> expressive enough to handle the cases above. Whether the cases solve >> actual problems or not, I couldn't say, but they seem like reasonable >> cases. >> >> It would, of course, be nicest if OS metadata had been invented way >> back when, for all OSes, such that all text files were flagged with >> their encoding... then languages could just read the encoding and do >> the right thing! But we live in the real world, instead. >> > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? That sounds very flexible -- but in net effect it would only make illegal a subset of the BOM-containing encodings (those not listed) without making legal any additional encodings other than the non-BOM encoding. Whether prohibiting a subset of BOM-containing encodings is a useful use case, I couldn't say... but my goal would be to included as many different file encodings on input as possible: without a BOM, that is exactly 1 (unless there are other heuristics), with a BOM, it is 1+all-BOM-containing encodings. Your scheme would permit numbers of encodings accepted to vary between 1 and 1+all-BOM-containing encodings. (I think everyone can agree there are 5 different byte sequences that can be called a Unicode BOM. The likelihood of them appearing in any other text encoding created by mankind depends on those other encodings -- but it is not impossible. It is truly up to the application to decide whether BOM detection could potentially conflict with files in some other encoding that would be acceptable to the application.) So I think it is taking it further than I can see value in, but I'm willing to be convinced otherwise. I see only a need for detecting BOM, and specifying a default encoding to be used if there is no BOM. Note that it might be nice to have a specification for using current encoding=None heuristic -- perhaps encoding="BOM-None" per my originally proposed syntax. But I'm still not saying that is the best syntax. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ncoghlan at gmail.com Sat Jan 9 04:07:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 09 Jan 2010 13:07:12 +1000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B476196.4080409@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B476196.4080409@mrabarnett.plus.com> Message-ID: <4B47F2E0.7090403@gmail.com> MRAB wrote: > Maybe there should also be a way of determining what encoding it decided > it was, so that you can then write a new file in that same encoding. I thought of that question as well - the f.encoding attribute on the opened file should be sufficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Sat Jan 9 06:48:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 06:48:36 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47E098.6030703@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> <4B47E098.6030703@g.nevcal.com> Message-ID: <319e029f1001082148h5cee2c0bn76f70a17766ca69e@mail.gmail.com> It seems to me that when opening a file, the following is the only flow that makes sense for the typical opening of a file flow: if encoding is not None: use encoding elif file has BOM: use BOM else: use system default And hence a encoding='BOM' isn't needed there. Although I'm trying to come up with usecases that doesn't work with this, I can't. :) BUT When writing things are not so easy though. Apparently some encodings require a BOM to be written, but others do not, but allow it, and some has no byte order mark. So there you have to be able to write the BOM, or not. And that's either a new parameter, because you can't use encoding='BOM' since you need to specify the encoding as well, or a new method. I would suggest a BOM parameter, and maybe a method as well. BOM=None|True|False Where "None" means a sane default behaviour, that is write a BOM if the encoding require it. "True" means write a BOM if the encoding *supports* it. "False" means Don't write a BOM even if the encoding requires it (because I know what I'm doing) if 'w' in mode: # But not 'r' or 'a' if BOM == True and encoding in (ENCODINGS THAT ALLOW BOM): write_bom = True elif BOM == False: write_bom = False elif BOM == None and encoding in (ENCODINGS THAT REQUIRE BOM): write_bom = True else: write_bom = False else: write_bom = False For reading this parameter could either be a noop, or possibly change the behavior somehow, if a usecase where that makes sense can be imagined. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From walter at livinglogic.de Sat Jan 9 11:51:57 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 11:51:57 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B485FCD.2040901@livinglogic.de> On 09.01.10 01:47, Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. This is doable with the currect API. Simply define a codec search function that handles all encoding names that start with "BOM-" and pass the "otherEncodingForDefault" part along to the codec. > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. Servus, Walter From walter at livinglogic.de Sat Jan 9 12:18:33 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 12:18:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081140.28124.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> Message-ID: <4B486609.2050804@livinglogic.de> Victor Stinner wrote: > Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : >>> Builtin open() function is unable to open an UTF-16/32 file starting with >>> a BOM if the encoding is not specified (raise an unicode error). For an >>> UTF-8 file starting with a BOM, read()/readline() returns also the BOM >>> whereas the BOM should be "ignored". >> It depends. If you use the utf-8-sig encoding, it *will* ignore the >> UTF-8 signature. > > Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and > UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to > remove the BOM after the first read (much harder if you use a module like > ConfigParser or csv). > >>> Since my proposition changes the result TextIOWrapper.read()/readline() >>> for files starting with a BOM, we might introduce an option to open() to >>> enable the new behaviour. But is it really needed to keep the backward >>> compatibility? >> Absolutely. And there is no need to produce a new option, but instead >> use the existing options: define an encoding that auto-detects the >> encoding from the family of BOMs. Maybe you call it encoding="sniff". > > Good idea, I choosed open(filename, encoding="BOM"). On the surface this looks like there's an encoding named "BOM", but looking at your patch I found that the check is still done in TextIOWrapper. IMHO the best approach would to the implement a *real* codec named "BOM" (or "sniff"). This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop. To see how something like this can be done, take a look at the UTF-16 codec, that switches to bigendian or littleendian mode depending on the first read/decode call. Servus, Walter From victor.stinner at haypocalc.com Sat Jan 9 13:37:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:37:06 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47DA7B.6050505@v.loewis.de> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> <4B47DA7B.6050505@v.loewis.de> Message-ID: <201001091337.06596.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:23:07, Martin v. L?wis a ?crit : > While I would support combining BOM detection in the case where a file > is opened for reading and no encoding is specified, I see two problems: > a) if a seek operations is performed before having looked at the BOM, > no determination would have been made TextIOWrapper doesn't support seek to an arbitrary byte. It uses "cookie" which is an opaque value. Reuse a cookie from another file or an old cookie is forbidden (but it doesn't raise an error). This is not specific to the BOM checking: the problem already exist for encodings using a BOM (eg. UTF-16). > b) what encoding should it use on writing? Don't change anything to writing. With Antoince choice: open('file.txt', 'w', encoding=None) continue to use the actual heuristic (os.device_encoding() or system locale). With Guido choice, encoding="BOM": it raises an error, because BOM check is not supported when writing into a file. How could the BOM be checked when creating a new (empty) file!? -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Sat Jan 9 13:45:58 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 09 Jan 2010 13:45:58 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B487A86.4020603@egenix.com> Victor Stinner wrote: > (2) Check for a BOM while reading or detect it before? > > Everybody agree that checking BOM is an interesting option and should not be > limited to open(). > > Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file > name or a binary file-like object: it returns the encoding and seek to the > file start or just after the BOM. > > I dislike this function because it requires extra file operations (open > (optional), read() and seek()) and it doesn't work if the file is not seekable > (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to > avoid extra file operations. > > Note: I implemented the BOM check in TextIOWrapper; so it's already usable for > any file-like object. Yes, but the implementation is limited to just BOM checking and thus only supports UTF-8-SIG, UTF-16 and UTF-32. With a codecs module function we could easily extend the encoding detection to more file types, e.g. XML files, Python source code files, etc. that use other mechanisms for defining the encoding. BTW: I haven't looked at your implementation, but what happens when your BOM check fails ? Will the implementation add the already read bytes back to a buffer ? This rollback action is the only reason for needing a seekable stream in codecs.guess_stream_encoding(). Another point to consider: AFAIK, we currently have a moratorium on changes to Python builtins. How does that match up with the proposed changes ? Using a new codec like Walter suggested would move the implementation into the stdlib for which doesn't the moratorium doesn't apply. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Sat Jan 9 13:54:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:54:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <201001091354.17239.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 01:47:38, vous avez ?crit : > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. If no BOM is found, it fallback to the current heuristic: os.device_encoding() or system local. > (...) Hence, it might be that someone would expect a UTF-16LE (or any of > the formats that don't require a BOM, rather than UTF-8), but be willing > to accept any BOM-discriminated format. > (...) declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted You mean "if there is a BOM, use it, otherwise fallback to a specific charset"? How could it be declared? Maybe: open("file.txt", check_bom=True, encoding="UTF16-LE") open("file.txt", check_bom=True, encoding="latin1") About falling back to UTF-8, it would be written: open("file.txt", check_bom=True, encoding="UTF-8") As explained before, check_bom=True is only accepted for read only file mode. Well, why not. This is a third choice for my point (1) :-) It's between Guido and Antoine choice, and I like it because we can fallback to UTF-8 instead of the dummy system locale: Windows users will be happy to be able to use UTF-8 :-) I prefer to fallback to a fixed encoding then depending on the system locale. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:34:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:34:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <201001091434.17521.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:12:28, MRAB a ?crit : > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? Yes, you're taking it foo far :-) Checking BOM is reliable, whereas *guessing* the charset only using the byte stream can only be an heuristic. Guess a charset is a complex problem, they are 3rd party library to do that, like the chardet project. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:38:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:38:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B486609.2050804@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <201001091438.43576.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : > > Good idea, I choosed open(filename, encoding="BOM"). > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Why not, this is another solution to the point (2) (Check for a BOM while reading or detect it before?). Which encoding would be used if there is not BOM? UTF-8 sounds like a good choice. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:50:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:50:28 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B487A86.4020603@egenix.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B487A86.4020603@egenix.com> Message-ID: <201001091450.28497.victor.stinner@haypocalc.com> Hi, Le samedi 09 janvier 2010 13:45:58, vous avez ?crit : > > Note: I implemented the BOM check in TextIOWrapper; so it's already > > usable for any file-like object. > > Yes, but the implementation is limited to just BOM checking > and thus only supports UTF-8-SIG, UTF-16 and UTF-32. Sure, but that's already better than no BOM check :-) It looks like many people would apprecite UTF-8-SIG detection, since this encoding is common on Windows. > BTW: I haven't looked at your implementation, but what happens > when your BOM check fails ? Will the implementation add the > already read bytes back to a buffer ? My implementation is done between buffer.read() and decoder.decode(data). If there is a BOM: set the encoding and remove the BOM bytes from the byte string. Otherwise, use another algorithm to choose the encoding and leave the byte string unchanged. It can be seen as a codec: it works like UTF-16 and UTF-32 codecs ;-) > AFAIK, we currently have a moratorium on changes to Python > builtins. How does that match up with the proposed changes ? Oh yes, I forgot the moratorium. In all solutions, some of them don't change the API. Eg. Antoine proposed to leave the API unchanged: open(file) => open(file) :-) I don't know if it's compatible with the moratorium or not. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 16:05:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 15:05:45 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: Walter D?rwald livinglogic.de> writes: > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Sorry but this is missing the point. The point here is to improve the open() function. I'm sure people who know about encodings are able to install the chardet library or even whip up their own BOM detection routine... Antoine. From benjamin at python.org Sat Jan 9 18:29:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 11:29:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 Message-ID: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> On behalf of the Python development team, I'm gleeful to announce the second alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From kmtracey at gmail.com Sat Jan 9 19:48:12 2010 From: kmtracey at gmail.com (Karen Tracey) Date: Sat, 9 Jan 2010 13:48:12 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce the > second > alpha release of Python 2.7. > > Well yay. Django's test suite (1242 tests) runs with just one failure on the 2.7 alpha 2 level, and that looks to be likely due to the improved string/float rounding so not really a problem, just a difference. That's down from 104 failures and 40 errors with 2.7 alpha 1. Note on the website page http://www.python.org/download/releases/2.7/ the "Change log for this release" link is still pointing to the alpha 1 changelog. Thanks, Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jan 9 19:51:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 12:51:11 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <1afaf6161001091051kb1ba08q9b96094af16c12fa@mail.gmail.com> 2010/1/9 Karen Tracey : > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson > wrote: >> >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> > > Well yay.? Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference.? That's > down from 104 failures and 40 errors with 2.7 alpha 1. Excellent! > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Thanks. I'll fix that. > -- Regards, Benjamin From skip at pobox.com Sat Jan 9 21:00:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:00:44 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 Message-ID: <19272.57452.972234.643008@montanaro.dyndns.org> How much of the Unladen Swallow cPickle speedups have been incorporated into 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the two versions I currently care about at work - we will skip 2.5 entirely). It appears some of their speedups may have already been merged to trunk, but I'm not sure how much. If a patch to merge this to 2.7 is already under consideration I won't look at it, but I interpreted Collin Winter's response to my query on the u-s mailing list that not everything has been done yet. Thx, Skip From martin at v.loewis.de Sat Jan 9 21:09:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 09 Jan 2010 21:09:27 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <4B48E277.9010408@v.loewis.de> Antoine Pitrou wrote: > Walter D?rwald livinglogic.de> writes: >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Sorry but this is missing the point. The point here is to improve the open() > function. I'm sure people who know about encodings are able to install the > chardet library or even whip up their own BOM detection routine... How does the requirement that it be implemented as a codec miss the point? FWIW, I agree with Walter that if it is provided through the encoding= argument, it should be a codec. If it is built into the open function (for whatever reason), it must be provided by some other parameter. I do see the point that it becomes available to end users only when released as part of Python. However, this *also* means that applications won't be using it for another three years or so, since they'll have to support older Python versions as well (unless it is integrated in the case where no encoding is specified). Regards, Martin From pjenvey at underboss.org Sat Jan 9 21:09:29 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 9 Jan 2010 12:09:29 -0800 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <19272.57452.972234.643008@montanaro.dyndns.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> On Jan 9, 2010, at 12:00 PM, skip at pobox.com wrote: > How much of the Unladen Swallow cPickle speedups have been incorporated into > 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the > two versions I currently care about at work - we will skip 2.5 entirely). > It appears some of their speedups may have already been merged to trunk, but > I'm not sure how much. If a patch to merge this to 2.7 is already under > consideration I won't look at it, but I interpreted Collin Winter's response > to my query on the u-s mailing list that not everything has been done yet. They've documented their upstream patches here: http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches -- Philip Jenvey From skip at pobox.com Sat Jan 9 21:21:20 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:21:20 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> Message-ID: <19272.58688.173011.412712@montanaro.dyndns.org> Philip> They've documented their upstream patches here: Philip> http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches Thanks. That will help immensely. Skip From solipsis at pitrou.net Sat Jan 9 21:28:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:28:12 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Sorry but this is missing the point. The point here is to improve the open() > > function. I'm sure people who know about encodings are able to install the > > chardet library or even whip up their own BOM detection routine... > > How does the requirement that it be implemented as a codec miss the > point? If we want it to be the default, it must be able to fallback on the current locale-based algorithm if no BOM is found. I don't think it would be easy for a codec to do that. > FWIW, I agree with Walter that if it is provided through the encoding= > argument, it should be a codec. If it is built into the open function > (for whatever reason), it must be provided by some other parameter. Why not simply encoding=None? The default value should provide the most useful behaviour possible. Forcing users to choose between two different autodetection strategies (encoding=None and another one) is a little insane IMO. Regards Antoine. From solipsis at pitrou.net Sat Jan 9 21:32:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:32:27 +0000 (UTC) Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > If a patch to merge this to 2.7 is already under > consideration I won't look at it, Why won't you look at it? :) Actually, if these patches are to be merged someone should certainly look at them, and do the (possibly) remaining work. http://bugs.python.org/issue5683 http://bugs.python.org/issue5671 Thank you Antoine. From skip at pobox.com Sat Jan 9 21:43:42 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:43:42 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <19272.60030.25319.269597@montanaro.dyndns.org> >>>>> "Antoine" == Antoine Pitrou writes: Antoine> pobox.com> writes: >> >> If a patch to merge this to 2.7 is already under >> consideration I won't look at it, Antoine> Why won't you look at it? :) I meant I wouldn't look at developing one. Skip From regebro at gmail.com Sat Jan 9 23:14:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 23:14:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Right. It seems like encoding=None is the right way to go there. encoding='BOM' would probably only work if 'BOM' isn't an encoding but a special tag, which is ugly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Sun Jan 10 00:25:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 23:25:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> Message-ID: <4B49105E.303@voidspace.org.uk> On 09/01/2010 22:14, Lennart Regebro wrote: > On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. >> > Right. It seems like encoding=None is the right way to go there. > encoding='BOM' would probably only work if 'BOM' isn't an encoding but > a special tag, which is ugly. > > I would rather see it as the default behavior for open without an encoding specified. I know Guido has expressed a preference against this so I won't continue to flog it. The current behavior however is that we have a 'guessing' algorithm based on the platform default. Currently if you open a text file in read mode that has a UTF-8 signature, but the platform default is something other than UTF-8, then we open the file using what is likely to be the incorrect encoding. Looking for the signature seems to be better behaviour in that case. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jan 10 00:40:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 10 Jan 2010 00:40:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <4B4913DF.7050801@v.loewis.de> >> How does the requirement that it be implemented as a codec miss the >> point? > > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Yes - however, Victor currently apparently *doesn't* want it to be the default, but wants the user to specify encoding="BOM". If so, it isn't the default, and it is easy to implement as a codec. >> FWIW, I agree with Walter that if it is provided through the encoding= >> argument, it should be a codec. If it is built into the open function >> (for whatever reason), it must be provided by some other parameter. > > Why not simply encoding=None? I don't mind. Please re-read Walter's message - it only said that *if* this is activated through encoding="BOM", *then* it must be a codec, and could be on PyPI. I don't think Walter was talking about the case "it is not activated through encoding='BOM'" *at all*. > The default value should provide the most useful > behaviour possible. Forcing users to choose between two different autodetection > strategies (encoding=None and another one) is a little insane IMO. That wouldn't disturb me much. There are a lot of things in that area that are a little insane, starting with Microsoft Windows :-) Regards, Martin From henning.vonbargen at arcor.de Sun Jan 10 12:10:02 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Sun, 10 Jan 2010 12:10:02 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: Message-ID: <4B49B58A.4040103@arcor.de> If Python should support BOM when reading text files, it should also be able to *write* such files. An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what? That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values: None: default (old) behaviour: don't handle BOM at all True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument. (*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False. Henning P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway... From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brett at python.org Sun Jan 10 19:51:26 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 10:51:26 -0800 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: Nick Coghlan thought I should forward this to python-dev so people are aware of this change and why it occurred (plus it indirectly informs Guido I finally finished the work). ---------- Forwarded message ---------- From: brett.cannon Date: Sat, Jan 9, 2010 at 18:56 Subject: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c To: python-checkins at python.org Author: brett.cannon Date: Sun Jan 10 03:56:19 2010 New Revision: 77402 Log: DeprecationWarning is now silent by default. This was originally suggested by Guido, discussed on the stdlib-sig mailing list, and given the OK by Guido directly to me. What this change essentially means is that Python has taken a policy of silencing warnings that are only of interest to developers by default. This should prevent users from seeing warnings which are triggered by an application being run against a new interpreter before the app developer has a chance to update their code. Closes issue #7319. Thanks to Antoine Pitrou, Ezio Melotti, and Brian Curtin for helping with the issue. Modified: python/trunk/Doc/library/warnings.rst python/trunk/Lib/test/test_ascii_formatd.py python/trunk/Lib/test/test_exceptions.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS python/trunk/Python/_warnings.c Modified: python/trunk/Doc/library/warnings.rst ============================================================================== --- python/trunk/Doc/library/warnings.rst (original) +++ python/trunk/Doc/library/warnings.rst Sun Jan 10 03:56:19 2010 @@ -59,7 +59,7 @@ | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,14 +151,6 @@ :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - .. _warning-suppress: @@ -226,6 +221,37 @@ entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions Modified: python/trunk/Lib/test/test_ascii_formatd.py ============================================================================== --- python/trunk/Lib/test/test_ascii_formatd.py (original) +++ python/trunk/Lib/test/test_ascii_formatd.py Sun Jan 10 03:56:19 2010 @@ -4,6 +4,7 @@ import unittest from test_support import check_warnings, run_unittest, cpython_only +import warnings class FormatDeprecationTests(unittest.TestCase): @@ -17,6 +18,7 @@ buf = create_string_buffer(' ' * 100) with check_warnings() as w: + warnings.simplefilter('default') PyOS_ascii_formatd(byref(buf), sizeof(buf), '%+.10f', c_double(10.0)) self.assertEqual(buf.value, '+10.0000000000') Modified: python/trunk/Lib/test/test_exceptions.py ============================================================================== --- python/trunk/Lib/test/test_exceptions.py (original) +++ python/trunk/Lib/test/test_exceptions.py Sun Jan 10 03:56:19 2010 @@ -309,6 +309,7 @@ # BaseException.__init__ triggers a deprecation warning. exc = BaseException("foo") with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('default') self.assertEquals(exc.message, "foo") self.assertEquals(len(w), 1) self.assertEquals(w[0].category, DeprecationWarning) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Sun Jan 10 03:56:19 2010 @@ -383,8 +383,8 @@ # Module initialization _processoptions(sys.warnoptions) if not _warnings_defaults: - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) + for cls in (DeprecationWarning, PendingDeprecationWarning, ImportWarning): + simplefilter("ignore", category=cls, append=True) bytes_warning = sys.flags.bytes_warning if bytes_warning > 1: bytes_action = "error" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Jan 10 03:56:19 2010 @@ -12,6 +12,8 @@ Core and Builtins ----------------- +- Issue #7319: Silence DeprecationWarning by default. + - Issue #2335: Backport set literals syntax from Python 3.x. Library Modified: python/trunk/Python/_warnings.c ============================================================================== --- python/trunk/Python/_warnings.c (original) +++ python/trunk/Python/_warnings.c Sun Jan 10 03:56:19 2010 @@ -85,10 +85,10 @@ default_action = get_warnings_attr("defaultaction"); if (default_action == NULL) { - if (PyErr_Occurred()) { - return NULL; - } - return _default_action; + if (PyErr_Occurred()) { + return NULL; + } + return _default_action; } Py_DECREF(_default_action); @@ -202,12 +202,12 @@ mod_str = PyString_AsString(filename); if (mod_str == NULL) - return NULL; + return NULL; len = PyString_Size(filename); if (len < 0) return NULL; if (len >= 3 && - strncmp(mod_str + (len - 3), ".py", 3) == 0) { + strncmp(mod_str + (len - 3), ".py", 3) == 0) { module = PyString_FromStringAndSize(mod_str, len-3); } else { @@ -251,7 +251,7 @@ name = PyObject_GetAttrString(category, "__name__"); if (name == NULL) /* XXX Can an object lack a '__name__' attribute? */ - return; + return; f_stderr = PySys_GetObject("stderr"); if (f_stderr == NULL) { @@ -341,7 +341,7 @@ rc = already_warned(registry, key, 0); if (rc == -1) goto cleanup; - else if (rc == 1) + else if (rc == 1) goto return_none; /* Else this warning hasn't been generated before. */ } @@ -492,9 +492,9 @@ /* Setup filename. */ *filename = PyDict_GetItemString(globals, "__file__"); if (*filename != NULL) { - Py_ssize_t len = PyString_Size(*filename); + Py_ssize_t len = PyString_Size(*filename); const char *file_str = PyString_AsString(*filename); - if (file_str == NULL || (len < 0 && PyErr_Occurred())) + if (file_str == NULL || (len < 0 && PyErr_Occurred())) goto handle_error; /* if filename.lower().endswith((".pyc", ".pyo")): */ @@ -506,10 +506,10 @@ tolower(file_str[len-1]) == 'o')) { *filename = PyString_FromStringAndSize(file_str, len-1); - if (*filename == NULL) - goto handle_error; - } - else + if (*filename == NULL) + goto handle_error; + } + else Py_INCREF(*filename); } else { @@ -536,8 +536,8 @@ else { /* embedded interpreters don't have sys.argv, see bug #839151 */ *filename = PyString_FromString("__main__"); - if (*filename == NULL) - goto handle_error; + if (*filename == NULL) + goto handle_error; } } if (*filename == NULL) { @@ -839,26 +839,29 @@ static PyObject * init_filters(void) { - PyObject *filters = PyList_New(3); + PyObject *filters = PyList_New(4); const char *bytes_action; if (filters == NULL) return NULL; PyList_SET_ITEM(filters, 0, + create_filter(PyExc_DeprecationWarning, "ignore")); + PyList_SET_ITEM(filters, 1, create_filter(PyExc_PendingDeprecationWarning, "ignore")); - PyList_SET_ITEM(filters, 1, create_filter(PyExc_ImportWarning, "ignore")); + PyList_SET_ITEM(filters, 2, create_filter(PyExc_ImportWarning, "ignore")); if (Py_BytesWarningFlag > 1) bytes_action = "error"; else if (Py_BytesWarningFlag) bytes_action = "default"; else bytes_action = "ignore"; - PyList_SET_ITEM(filters, 2, create_filter(PyExc_BytesWarning, + PyList_SET_ITEM(filters, 3, create_filter(PyExc_BytesWarning, bytes_action)); if (PyList_GET_ITEM(filters, 0) == NULL || PyList_GET_ITEM(filters, 1) == NULL || - PyList_GET_ITEM(filters, 2) == NULL) { + PyList_GET_ITEM(filters, 2) == NULL || + PyList_GET_ITEM(filters, 3) == NULL) { Py_DECREF(filters); return NULL; } _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sun Jan 10 20:22:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 10 Jan 2010 20:22:10 +0100 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: <201001102022.10259.victor.stinner@haypocalc.com> Hi, Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : > Nick Coghlan thought I should forward this to python-dev so people are > aware of this change and why it occurred (plus it indirectly informs Guido > I finally finished the work). Thanks to copy this mail to the python-dev mailing list. What is the recommanded way to get the previous behaviour (print deprecation warnings once)? Is there a command line option? Is it "-W default"? -- Victor Stinner http://www.haypocalc.com/ From benjamin at python.org Sun Jan 10 20:23:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 13:23:54 -0600 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <201001102022.10259.victor.stinner@haypocalc.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> <201001102022.10259.victor.stinner@haypocalc.com> Message-ID: <1afaf6161001101123g366d80dbjeaa80f1a632605e2@mail.gmail.com> 2010/1/10 Victor Stinner : > Hi, > > Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : >> Nick Coghlan thought I should forward this to python-dev so people are >> ?aware of this change and why it occurred (plus it indirectly informs Guido >> ?I finally finished the work). > > Thanks to copy this mail to the python-dev mailing list. > > What is the recommanded way to get the previous behaviour (print deprecation > warnings once)? Is there a command line option? Is it "-W default"? That commit conveniently adds documentation answering that question. :) -- Regards, Benjamin From nas at arctrix.com Sun Jan 10 20:30:09 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:30:09 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce > the second alpha release of Python 2.7. Thanks to everyone who contributed. > Python 2.7 is scheduled to be the last major version in the 2.x > series. Has this really been decided already? Maybe I missed it. In my opinion, it does Python's reputation harm to make such a statement. Conservative users (probably the vast majority of Python users) don't like to hear that software they are considering using is nearing the end of its life. What does it gain us to announce that the 2.x branch is dead aside from bugfixes? I propose that the 2.x branch be treated like 2.x.y branches: as long as there is sufficient volunteer labour, it should continue to live. In order to avoid wasted development effort, it would be prudent to announce that unless a 2.8 release manager steps up, whatever is committed to the 2.x branch after the 2.7 release may never get released. Said another way, it's okay for the Python developers to decide to abandon 2.x and put their efforts into 3.x. It's not okay for them to prevent others from continuing to work on 2.x or to somehow make 2.x look worse so 3.x looks better. Python 3 needs to stand on its own terms and I'm confident it can. Best regards, Neil From brett at python.org Sun Jan 10 21:09:08 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:09:08 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sun, Jan 10, 2010 at 11:30, Neil Schemenauer wrote: > Benjamin Peterson wrote: > > On behalf of the Python development team, I'm gleeful to announce > > the second alpha release of Python 2.7. > > Thanks to everyone who contributed. > > > Python 2.7 is scheduled to be the last major version in the 2.x > > series. > > Has this really been decided already? Maybe I missed it. More or less. It was first discussed at the language summit last year and has come up here a couple of times. If needed we can make it official in terms of lifetime of 2.7, etc. at the language summit this year. > In my > opinion, it does Python's reputation harm to make such a statement. > Conservative users (probably the vast majority of Python users) > don't like to hear that software they are considering using is > nearing the end of its life. What does it gain us to announce that > the 2.x branch is dead aside from bugfixes? > > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. > > Said another way, it's okay for the Python developers to decide to > abandon 2.x and put their efforts into 3.x. It's not okay for them > to prevent others from continuing to work on 2.x or to somehow make > 2.x look worse so 3.x looks better. Python 3 needs to stand on its > own terms and I'm confident it can. > I don't think ending the 2.x series at 2.7 makes it look bad compared to 3.2; it's simply the end of a development line like any other software project. I suspect 2.7 will have a protracted bugfix window because so much code runs on 2.x exclusively at the moment. And if core developers want to continue to backport fixes past two years from release they can (or however long we decide to officially support 2.7). No one is saying people still can't work on the code, just that python-dev as an entity is not going to focus its effort on the 2.x series anymore and people should not rely upon us to continue to focus new development efforts in that series. If there are core developers who want to continue to do bugfix releases then that's fine and I am happy to flag patches as needing backports and let other's do the work after the standard two year maintenance cycle, but I know I do not want to be held accountable as a core developer to keep the 2.x going indefinitely. Maintaining four branches is enough of a reason in my book to not keep the 2.x series going. If there really is an outcry on this we can re-visit the issue, but as of right now we need to move forward at some point and 2.7 seems like that good point. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 10 21:23:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Jan 2010 06:23:27 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A373F.9050601@gmail.com> Neil Schemenauer wrote: > In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. The announcement is precisely to avoid the situation where people commit new features to the 2.x main line of development (after the 2.7 maintenance branch is created) in the expectation that they will be released as part of a hypothetical 2.8 release. Whether that info needs to be in each and every 2.7 announcement... probably not. It isn't really info for users of Python, just for developers of Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jan 10 21:25:29 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:25:29 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit Message-ID: Michael has given me the hg transition/stdlib time slot at the language summit this year. In regards to that I plan to lead a discussion on: * where we are at w/ the Hg transition (Dirkjan should be there and I did a blog post on this topic recently: http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) * argparse (PEP 389) * brief mention on still wanting to break out the stdlib from CPython * an official policy on extension modules? (i.e. must have a pure Python implementation which can mean a ctypes implementation unless given an explicit waiver) That's everything from a stdlib perspective. I have half-baked ideas, but nothing concrete enough to bring up unless people really want to discuss stuff like how to potentially standardize module deprecation warnings, etc. In terms of me personally, I do plan to bring up at some point during the summit these points which don't squarely fit during my time slot: * an official unofficial policy on how new proposed features should come to us (i.e. working code to python-ideas with a shell of a PEP that includes abstract and motivation -> hashed out PEP to python-dev -> pronouncement) * any changes needed to the issue tracker to help with the workflow? (stage field seems like a failed experiment and we now have several effective triage people who can help w/ guiding changes) If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Sun Jan 10 22:52:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 10 Jan 2010 22:52:00 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) Unfortunately my flight doesn't land until about the time the Mercurial slot ends, so while I'll be there later on that afternoon for sprinting or questions or anything, I won't be at the actual Mercurial talk at the summit. Cheers, Dirkjan From fuzzyman at voidspace.org.uk Sun Jan 10 23:27:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 10 Jan 2010 22:27:58 +0000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4A546E.8010808@voidspace.org.uk> On 10/01/2010 21:52, Dirkjan Ochtman wrote: > On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > >> Michael has given me the hg transition/stdlib time slot at the language >> summit this year. In regards to that I plan to lead a discussion on: >> * where we are at w/ the Hg transition (Dirkjan should be there and I did a >> blog post on this topic recently: >> http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) >> > Unfortunately my flight doesn't land until about the time the > Mercurial slot ends, so while I'll be there later on that afternoon > for sprinting or questions or anything, I won't be at the actual > Mercurial talk at the summit. > > We will probably be in 'open discussion' when you arrive so it will still be good to hear from you on the topic and there maybe questions that can't be answered until you arrive... :-) Michael > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Mon Jan 11 00:02:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:02:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A5C69.7090007@v.loewis.de> > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. I think it's more difficult than that. It also takes developer effort to *commit* to the trunk. So if the trunk continued to live, would it still be ok if I closed a 2.x feature request as "won't fix", or only committed the new feature to the 3.x branch? As for old branches: they *don't* live in the way you claim (i.e. remain open with changes potentially just not released). Instead, at some point, they are frozen to bug-fix only, then to security-fix only, and then to no change at all. Regards, Martin From martin at v.loewis.de Mon Jan 11 00:07:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:07:58 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A373F.9050601@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> Message-ID: <4B4A5DCE.3070909@v.loewis.de> > The announcement is precisely to avoid the situation where people commit > new features to the 2.x main line of development (after the 2.7 > maintenance branch is created) in the expectation that they will be > released as part of a hypothetical 2.8 release. > > Whether that info needs to be in each and every 2.7 announcement... > probably not. It isn't really info for users of Python, just for > developers of Python. I think the announcement is indeed also for the users, and I would prefer it to stay in the release announcements, so that people will know for sure (and speak up) before the 2.7 release happens. As for decisions: I don't think there was an official BDFL pronouncent, but I recall Guido posting a message close to that, proposing that 2.7 will be a release that will see bug fix releases for an indefinite period of time (where indefinite != infinite). This was shortly after him proposing that perhaps we shouldn't make a 2.7 release at all, and stop at 2.6. As for such a decision giving a bad light on Python: I don't think that will be the case. Instead, I hear many users surprised for how long we have been maintaining to parallel versions - "that must have taken a lot of effort". So everybody will likely understand that enough is enough. Regards, Martin From benjamin at python.org Mon Jan 11 01:07:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 18:07:35 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Consider this program: class Descr(object): def __init__(self, name): self.name = name def __set__(self, instance, what): instance.__dict__[self.name] = what class X(object): attr = Descr("attr") x = X() print(x.attr) x.attr = 42 print(x.attr) It gives in output: <__main__.Descr object at 0x7fe1c9b28150> 42 The documentation [1] says that Descr is a data descriptor because it defines the __set__ method. It also states that data descriptors always override the value in the instance dictionary. So, the second line should also be the descriptor object according to the documentation. My question is: Is this a doc bug or a implementation bug? If the former, it will be the description of a data descriptor much less consistent, since it will require that a __get__ method be present, too. If the latter, the fix may break some programs relying on the ability to "cache" a value in the instance dictionary. [1] http://docs.python.org/reference/datamodel#invoking-descriptors -- Regards, Benjamin From amauryfa at gmail.com Mon Jan 11 01:51:09 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Jan 2010 01:51:09 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: Hi, 2010/1/11 Benjamin Peterson : > Consider this program: > > class Descr(object): > ? ?def __init__(self, name): > ? ? ? ?self.name = name > ? ?def __set__(self, instance, what): > ? ? ? ?instance.__dict__[self.name] = what > > class X(object): > ? ?attr = Descr("attr") > > x = X() > print(x.attr) > x.attr = 42 > print(x.attr) > > It gives in output: > > <__main__.Descr object at 0x7fe1c9b28150> > 42 > > The documentation [1] says that Descr is a data descriptor because it > defines the __set__ method. It also states that data descriptors > always override the value in the instance dictionary. So, the second > line should also be the descriptor object according to the > documentation. > > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors Quoting the documentation: """Normally, data descriptors define both __get__() and __set__(), while non-data descriptors have just the __get__() method. """ Your example is neither a data descriptor nor a non-data descriptor... The thing that worries me a bit is the "x.attr" returning the Descr object. Descriptors should remain at the class level, and instance should only see values. I'd prefer an AttributeError in this case. -- Amaury Forgeot d'Arc From benjamin at python.org Mon Jan 11 02:00:32 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 19:00:32 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <1afaf6161001101700u21006708l2ef62bfa257e4ce7@mail.gmail.com> 2010/1/10 Amaury Forgeot d'Arc : > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... See the footnote: http://docs.python.org/reference/datamodel#id7 > > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. Far too late for that, I'm afraid. -- Regards, Benjamin From nas at arctrix.com Mon Jan 11 02:44:57 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:44:57 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <20100111014457.GA5289@arctrix.com> On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > I don't think ending the 2.x series at 2.7 makes it look bad > compared to 3.2; it's simply the end of a development line like > any other software project. I suspect 2.7 will have a protracted > bugfix window because so much code runs on 2.x exclusively at the > moment. I would guess over 99% of all Python code written doesn't run on Python 3. Given that, I think it is premature to close the door on new major versions of Python 2.x. Also, we as a project should be careful not to present the image that Python 2.x will not be supported in the future. > If there really is an outcry on this we can re-visit the issue, > but as of right now we need to move forward at some point and 2.7 > seems like that good point. I think that's bad PR. If I had a successful product, I would not announce its end of life just to see how many customers scream and then decide if I should devote more resources to continue maintaining it. IMHO, the release notes should say something like: After the Python 2.7 release, the focus of Python development will be on Python 3. There will continue to be maintainance releases of Python 2.x. trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil From tjreedy at udel.edu Mon Jan 11 03:26:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Jan 2010 21:26:34 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On 1/10/2010 8:44 PM, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: >> I don't think ending the 2.x series at 2.7 makes it look bad >> compared to 3.2; it's simply the end of a development line like >> any other software project. I suspect 2.7 will have a protracted >> bugfix window because so much code runs on 2.x exclusively at the >> moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. If the removal of old features had been done in the 2.x series, as once planned (Guido originally proposed removing the old meaning of int / int in 2.5) the same more or less would be true of 2.7. It is past time for other old and now duplicated features to be removed also. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > >> If there really is an outcry on this we can re-visit the issue, >> but as of right now we need to move forward at some point and 2.7 >> seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. Python is not being ended, but upgraded (with bloating duplications removed). Think of 3.1 as 2.8 with a new name. tjr From lrekucki at gmail.com Mon Jan 11 04:26:48 2010 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Mon, 11 Jan 2010 04:26:48 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: > Date: Mon, 11 Jan 2010 01:51:09 +0100 > From: "Amaury Forgeot d'Arc" > To: Benjamin Peterson > Cc: Python Dev > Subject: Re: [Python-Dev] Data descriptor doc/implementation > ? ? ? ?inconsistency > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > 2010/1/11 Benjamin Peterson : >> Consider this program: >> >> class Descr(object): >> ? ?def __init__(self, name): >> ? ? ? ?self.name = name >> ? ?def __set__(self, instance, what): >> ? ? ? ?instance.__dict__[self.name] = what >> >> class X(object): >> ? ?attr = Descr("attr") >> >> x = X() >> print(x.attr) >> x.attr = 42 >> print(x.attr) >> >> It gives in output: >> >> <__main__.Descr object at 0x7fe1c9b28150> >> 42 >> >> The documentation [1] says that Descr is a data descriptor because it >> defines the __set__ method. It also states that data descriptors >> always override the value in the instance dictionary. So, the second >> line should also be the descriptor object according to the >> documentation. >> >> My question is: Is this a doc bug or a implementation bug? If the >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors > > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... Actually, there is this footnote [1]: """ A descriptor can define any combination of __get__(), __set__() and __delete__(). If it does not define __get__(), then accessing the attribute even on an instance will return the descriptor object itself. If the descriptor defines __set__() and/or __delete__(), it is a data descriptor; if it defines neither, it is a non-data descriptor. """ Which would mean Descr is actually a data descriptor without a __get__(), so x.attr should always return the descriptor object itself (at least in the docs). > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. > > -- > Amaury Forgeot d'Arc [1] http://docs.python.org/reference/datamodel#id7 -- Lukasz Rekucki From brett at python.org Mon Jan 11 04:46:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 19:46:04 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > > I don't think ending the 2.x series at 2.7 makes it look bad > > compared to 3.2; it's simply the end of a development line like > > any other software project. I suspect 2.7 will have a protracted > > bugfix window because so much code runs on 2.x exclusively at the > > moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Well yeah; Python 3.0 is just over a year old with 3.1 -- the first robust 3.x release - is about six months old. From the beginning uptake was expected to take years, not months, so saying that 3.x is not popular enough seems premature. > Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > No one has said bugfixes will cease. > > > If there really is an outcry on this we can re-visit the issue, > > but as of right now we need to move forward at some point and 2.7 > > seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. > > I never said that I wanted to make this announcement specifically to provoke outcries. I said if there happened to be outcries we could possibly visit the issue again. Basically I was being considerate and trying to leave the door open to discuss things in the future even though I don't see the situation changing. > IMHO, the release notes should say something like: > > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. > No because that suggests new features will be coming to 2.x which is not going to happen. If you want to say there will be continual bugfix releases for 2.7 as is par the course for Python and that the number of bugfix releases might be more than normal then I am okay with that. > > > trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil > That came and went already a couple months ago when we discussed stopping at 2.6 instead of 2.7 (see the various threads at http://mail.python.org/pipermail/python-dev/2009-November/thread.html). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 05:05:07 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:05:07 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <20100111040507.GA7200@arctrix.com> On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > After the Python 2.7 release, the focus of Python development > > will be on Python 3. There will continue to be maintainance > > releases of Python 2.x. > > No because that suggests new features will be coming to 2.x which is not > going to happen. If you want to say there will be continual bugfix releases > for 2.7 as is par the course for Python and that the number of bugfix > releases might be more than normal then I am okay with that. Are you are saying that if someone steps up to merge the Unladen Swallow features into a 2.8 release and someone also steps up to cut the release that they will be prevented from doing so? Also, are you presuming to channel the BDFL or was this dicussed somewhere other than python-dev? Maybe I'm overreacting but I get the feeling that the larger and less active segment of the Python develpment team hasn't been involved in these decisions. Best regards, Neil From nas at arctrix.com Mon Jan 11 05:17:54 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:17:54 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5C69.7090007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> Message-ID: <20100111041754.GB7200@arctrix.com> On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: > [...] would it still be ok if I closed a 2.x feature request as > "won't fix", or only committed the new feature to the 3.x branch? Yes. Non-bugfix development on 2.x would optional (i.e. done by people who want to spend the time). Since language changes are now out (an initiative I completely support), the majority of non-bugfix changes would be optimizations and platform porting. Regards, Neil From brett at python.org Mon Jan 11 06:06:15 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 21:06:15 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111040507.GA7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 20:05, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > > After the Python 2.7 release, the focus of Python development > > > will be on Python 3. There will continue to be maintainance > > > releases of Python 2.x. > > > > No because that suggests new features will be coming to 2.x which is not > > going to happen. If you want to say there will be continual bugfix > releases > > for 2.7 as is par the course for Python and that the number of bugfix > > releases might be more than normal then I am okay with that. > > Are you are saying that if someone steps up to merge the Unladen > Swallow features into a 2.8 release and someone also steps up to cut > the release that they will be prevented from doing so? > > I honestly don't know, but it's a possibility just like any other new feature request. If people start taking the carrots we have added to 3.x and backporting them to keep the 2.x series alive you are essentially making the 3.x DOA by negating its benefits which I personally don't agree with. > Also, are you presuming to channel the BDFL or was this dicussed > somewhere other than python-dev? This was discussed; see the November 2009 threads. Guido actually suggested ending the 2.x branch at 2.6 until people spoke up about the amount of stuff already backported to 2.7 from 3.x because it was being assumed to be the end of the series to warrant keeping it to help transition to 2.7. > Maybe I'm overreacting but I get > the feeling that the larger and less active segment of the Python > develpment team hasn't been involved in these decisions. > This all came up in November from the 3rd through the 6th (four days) over a ton of email traffic. This was not a snap decision but a heated discussion where even Guido participated that culminated with the decision to stop at 2.7 for the 2.x series; this was not a smoked-filled room decision. I'm sorry if people missed it when they weren't looking, but python-dev is supposed to be the place to watch for this kind of stuff. I don't know how else to bring this stuff to people's attention short of also on python-committers, but developers are basically expected to be on both lists so I don't see where anyone did anything wrong in terms of informing developers. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 06:27:13 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 23:27:13 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: <20100111052713.GA8211@arctrix.com> On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > If people start taking the carrots we have added to 3.x and > backporting them to keep the 2.x series alive you are essentially > making the 3.x DOA by negating its benefits which I personally > don't agree with. I think we have got to the heart of our disagreement. Assume that some superhuman takes all the backwards compatible goodies from 3.x and merges them into 2.x. If that really would kill off Python 3 then I would proclaim it a project that deserved to die. Why should the backwards incompatible changes be endured if they cannot justify their existance by the benefit they provide? I guess I have more confidence in Python 3 than you do. I don't see why Python 2.x needs to be artificially limited so that Python 3 can benefit. Regards, Neil From brett at python.org Mon Jan 11 07:30:49 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 22:30:49 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 21:27, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. > > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. If that really would kill off Python 3 > then I would proclaim it a project that deserved to die. Why should > the backwards incompatible changes be endured if they cannot justify > their existance by the benefit they provide? > > When I said "carrots" I meant everything that makes Python 3 the best version of Python, including the backward-incompatible changes. I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. > I don't view it as artificial but simply a focusing of the development team on what has been determined as the future of Python by Guido and letting go of the past. IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying "Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out". That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Jan 11 08:08:14 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 08:08:14 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <319e029f1001102308p5de87cber2131f3aec1abc9f0@mail.gmail.com> On Mon, Jan 11, 2010 at 06:27, Neil Schemenauer wrote: > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. The superhumans of core developers pretty much already have. That's pretty much 2.7 you are talking about. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From chrism at plope.com Mon Jan 11 08:27:03 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 11 Jan 2010 02:27:03 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4AD2C7.1050703@plope.com> Brett Cannon wrote: > IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev > saying "Python 3 is the future, but we are keeping the old Python 2.x > around because we don't have *that* much faith in the future we have > laid out". That's poison to Python 3 by showing a lack of confidence in > the direction that the BDFL and python-dev as a group has chosen. Now I > could be wrong and there could actually be a large number of active > contributors who want to keep the 2.x series going, but based on the > discussion that occurred the last time this came up I believe the guys > who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort "Python 2". - C From martin at v.loewis.de Mon Jan 11 09:06:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:06:16 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: <4B4ADBF8.3030803@v.loewis.de> Neil Schemenauer wrote: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). I think *that* would give a very bad impression of Python. Depending whom you deal with, the new feature you want may or may not get added to the code base. Contributors would feel even more stranded than they do now, where it may take several years to get a patch reviewed, as you then could submit a patch, and pray that a comitter reviews it who believes in future 2.x releases. The point of setting policies is that it gives every user (contributors, committers, and "end-user" developers) a reliable foundation for expectations. Regards, Martin From arcriley at gmail.com Mon Jan 11 09:06:10 2010 From: arcriley at gmail.com (Arc Riley) Date: Mon, 11 Jan 2010 03:06:10 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AD2C7.1050703@plope.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4AD2C7.1050703@plope.com> Message-ID: after all these years, some people still run AmigaOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 11 09:11:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:11:25 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <4B4ADD2D.6070906@v.loewis.de> > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. Why that? It is a fact: 2.x will not be supported, in some future. Should we lie to users? > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. It shouldn't say that, because it wouldn't be true. Regards, Martin From martin at v.loewis.de Mon Jan 11 09:18:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:18:30 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4ADED6.5080207@v.loewis.de> > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. Because it takes too much time. Too much of my time, but apparently also too much of other people's time. Of course, the less active fraction of Python contributors may not notice, since they just chose to not contribute (which, of course, is fine). However, asking me to work twice as much as I want to on the project to keep two branches alive is just unfair. This has nothing to do with pushing 3.x, but all with managing available manpower and still providing quality software. Regards, Martin From regebro at gmail.com Mon Jan 11 09:42:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 09:42:51 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> This is what it says now: "Python 2.7 is scheduled to be the last major version in the 2.x series before it moves into 5 years of bugfix-only mode. " And during this discussion it has been noted that others than the core python team might pick up Python 2 and make releases. So maybe we can and this discussion by changing that line in future releases to be: "Python 2.7 is scheduled to be the last major version in the 2.x series released by the Python Software Foundation before it moves into 5 years of bugfix-only mode. " That doesn't exclude *others* making a Python 2.8 that could include all sorts of crazy features. :) This is mainly just to get the pointless discussion over with. The current Python core team don't want to make a 2.8, so that will not happen. Someone else might, but as Chris points out, they won't step up until they have to, which is probably at least two years from now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 10:06:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 10:06:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> Message-ID: <4B4AEA25.7010100@v.loewis.de> > "Python 2.7 is scheduled to be the last major version in the 2.x > series released by the Python Software Foundation before it moves into > 5 years of bugfix-only mode. " > > That doesn't exclude *others* making a Python 2.8 that could include > all sorts of crazy features. :) > > This is mainly just to get the pointless discussion over with. I know you are just looking for a compromise, but this shouldn't be it: the PSF has deliberately stayed out of the actual Python engineering, so the release that Benjamin makes is not done by the PSF (but by Benjamin and his contributors :-). I like the wording as it is (although I find the promise of five years of bug fix releases a bit too strong). Regards, Martin From regebro at gmail.com Mon Jan 11 10:19:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 10:19:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AEA25.7010100@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> Message-ID: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" wrote: > I know you are just looking for a compromise, but this shouldn't be it: > the PSF has deliberately stayed out of the actual Python engineering, > so the release that Benjamin makes is not done by the PSF (but by > Benjamin and his contributors :-). Hm. Yeah. That's right of course. I started with saying "official", but then I thought "official according to who?". :) So I changed it to mentioning PSF, but that doesn't work either. I guess the current writing as as good as it gets, unless we change "scheduled" to "expected" or something. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Mon Jan 11 10:42:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:42:05 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: Neil Schemenauer, 11.01.2010 05:17: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). Note that there's also the time it takes to make a release (usually involving at least one beta release), plus the possibility of introducing bugs while adding new features or even while fixing agreed bugs. All of this takes time in addition to the time people might want to invest in 'real' development on the 2.x trunk. Stefan From stephen at xemacs.org Mon Jan 11 10:59:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Jan 2010 18:59:57 +0900 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <8763793qsi.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Schemenauer writes: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. Well, I think it's *worse* than that, and I don't think you really mean "DOA", anyway. (Feel free to correct me, of course.) The problem I see with backporting lots of stuff, and/or adding new features that aren't in 3.0, to 2.x is that it will make 2.x even cruftier, when it was already crufty enough that Guido (and almost all of python-dev) bit the bullet and said "backward compatibility is no excuse for keeping something in 3.0". That surely means that a lot of python-dev denizens will declare non-support 2.x for x > 7. It's not going to be the gradual migration we've seen over the past few months as active people start to spend more and more time on 3 vs. 2; it will be a watershed. Especially if these are new features merged from outside that the "small active segment" doesn't know anything about. From the users' point of view, that amounts to a *fork*, even if it's internal and "friendly". > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. Isn't that a bit ridiculous? I just don't see any evidence that anything like that is going to happen. Worse, if we *assume* it will happen, I don't see any way to assess whether (1) Python 3 goes belly up, (2) there's an effective fork confusing the users and draining the energy of python-dev, or (3) everybody goes "wow!" because it's so cool that everybody wants to keep maintaining an extra 3 branches indefinitely. My opinion is that given the clear direction the "small active segment" is going, telling the users anything but what Brett proposed is disinformation. > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. It's not for Python 3, which you, I, and I'm pretty sure Brett-in-his- heart-of-hearts agree can take care of itself because it is *better* than Python 2. It's for Python, and for the Python community. From walter at livinglogic.de Mon Jan 11 11:37:56 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 11:37:56 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001091438.43576.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> Message-ID: <4B4AFF84.7070206@livinglogic.de> On 09.01.10 14:38, Victor Stinner wrote: > Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : >>> Good idea, I choosed open(filename, encoding="BOM"). >> >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Why not, this is another solution to the point (2) (Check for a BOM while > reading or detect it before?). Which encoding would be used if there is not > BOM? UTF-8 sounds like a good choice. UTF-8 might be a good choice, are the failback could be specified in the encoding name, i.e. open("file.txt", encoding="BOM-UTF-8") falls back to UTF-8, if there's no BOM at the start. This could be implemented via a custom codec search function (see http://docs.python.org/library/codecs.html#codecs.register for more info). Servus, Walter From regebro at gmail.com Mon Jan 11 12:12:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:12:20 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4AFF84.7070206@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> Message-ID: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: > UTF-8 might be a good choice No, fallback if there is no BOM should be the local settings, just as fallback is today if you don't specify a codec. I mean, what if you want to look for a BOM but fall back to something else? How far will we go with encoding special information in the codecs names? codec='BOM else UTF-16 else locale'? :-) BOM is not a locale, and should not be a locale. Having a locale called BOM is wrong per se. It should either be default to look for a BOM when codec=None, or a special parameter. If none of these are desired, then we need a special function that takes a filename or file handle, and looks for a BOM and returns the codec found or None. But I find that much less natural and obvious than checking the BOM when codec=None. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Mon Jan 11 12:13:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:13:00 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <319e029f1001110313u1d839deodfe06a6c7ec423cf@mail.gmail.com> On Mon, Jan 11, 2010 at 12:12, Lennart Regebro wrote: > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. D'oh! I mean codec here obviously. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 13:29:04 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 13:29:04 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4913DF.7050801@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> Message-ID: <4B4B1990.90705@livinglogic.de> On 10.01.10 00:40, "Martin v. L?wis" wrote: >>> How does the requirement that it be implemented as a codec miss the >>> point? >> >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. > > Yes - however, Victor currently apparently *doesn't* want it to be the > default, but wants the user to specify encoding="BOM". If so, it isn't > the default, and it is easy to implement as a codec. > >>> FWIW, I agree with Walter that if it is provided through the encoding= >>> argument, it should be a codec. If it is built into the open function >>> (for whatever reason), it must be provided by some other parameter. >> >> Why not simply encoding=None? > > I don't mind. Please re-read Walter's message - it only said that > *if* this is activated through encoding="BOM", *then* it must be > a codec, and could be on PyPI. I don't think Walter was talking about > the case "it is not activated through encoding='BOM'" *at all*. However if this autodetection feature is useful in other cases (no matter how it's activated), it should be a codec, because as part of the open() function it isn't reusable. >> The default value should provide the most useful >> behaviour possible. Forcing users to choose between two different autodetection >> strategies (encoding=None and another one) is a little insane IMO. And encoding="mbcs" is a third option on Windows. > That wouldn't disturb me much. There are a lot of things in that area > that are a little insane, starting with Microsoft Windows :-) ;) Servus, Walter From barry at python.org Mon Jan 11 13:37:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 07:37:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5DCE.3070909@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: On Jan 10, 2010, at 6:07 PM, Martin v. L?wis wrote: > As for decisions: I don't think there was an official BDFL pronouncent, > but I recall Guido posting a message close to that, proposing that 2.7 > will be a release that will see bug fix releases for an indefinite > period of time (where indefinite != infinite). This was shortly after > him proposing that perhaps we shouldn't make a 2.7 release at all, and > stop at 2.6. IMO, the focus for Python 2.7 (and beyond) must be on helping people transition to Python 3. That should be the reason why a 2.7 is released and, what should dictate whether a 2.8 is necessary. Maybe everything people need (except manpower and round tuits) is already there. I was pleasantly surprised to find that Distribute supports automatically running 2to3 at install time for Python packages. This allowed me to "port" a recent (admittedly small) package to Python 3 by making it "python2.6 -3" clean and adding a couple of attributes to my setup.py[1]. I don't know how widely that feature's known, but I think it's *huge*, and exactly what we need to be doing. The more we can make it easy for people to port their Python 2 code to Python 3, and to support that with one code base, the quicker we'll get to a predominantly Python 3 world. So the question we should be asking is: what's missing from Python 2.7 to help with this transition? If we can't get it into 2.7, then why, and would pushing it back to 2.8 help at all? -Barry [1] modulo a bug in Distribute that caused doctest in separate files to not be used when running under Python 3. From solipsis at pitrou.net Mon Jan 11 13:39:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 13:39:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <1263213574.3327.0.camel@localhost> > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. It is reusable as part of io.TextIOWrapper, though. Regards Antoine. From regebro at gmail.com Mon Jan 11 13:45:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 13:45:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. But an autodetect feature is not a codec. Sure it should be reusable, but making it a codec seems to be a weird hack to me. And how would you reuse it if it was a codec? A reusable autodetect feature would be useable to detect what codec it is. A autodetect codec would not be useful for that, as it would simply just decode. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 14:21:15 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 14:21:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B25CB.5030003@livinglogic.de> On 11.01.10 13:45, Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: >> However if this autodetection feature is useful in other cases (no >> matter how it's activated), it should be a codec, because as part of the >> open() function it isn't reusable. > > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. I think we already had this discussion two years ago in the context of XML decoding ;): http://mail.python.org/pipermail/python-dev/2007-November/075138.html > And how would > you reuse it if it was a codec? A reusable autodetect feature would be > useable to detect what codec it is. A autodetect codec would not be > useful for that, as it would simply just decode. I have implemented an XML codec (as part of XIST: http://pypi.python.org/pypi/ll-xist), that can do that: >>> from ll import xml_codec >>> import codecs >>> c = codecs.getincrementaldecoder("xml")() >>> c.encoding >>> c.decode(">> c.encoding >>> c.decode(" version='1.0'") u'' >>> c.encoding >>> c.decode(" encoding='iso-8859-1'?>") u"" >>> c.encoding 'iso-8859-1' Servus, Walter From regebro at gmail.com Mon Jan 11 14:47:12 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 14:47:12 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B25CB.5030003@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B25CB.5030003@livinglogic.de> Message-ID: <319e029f1001110547n1d1c9831p34b0eac519d13f9d@mail.gmail.com> On Mon, Jan 11, 2010 at 14:21, Walter D?rwald wrote: > I think we already had this discussion two years ago in the context of > XML decoding ;): Yup. Ans Martins answer then is my answer now: "> So the code is good, if it is inside an XML parser, and it's bad if it > is inside a codec? Exactly so. This functionality just *isn't* a codec - there is no encoding. Instead, it is an algorithm for *detecting* an encoding." The conclusion was that a method do autodetect encodings would be good. I think the same conclusion applies here. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 18:16:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:16:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B5CF5.50806@v.loewis.de> > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. Well, the existing UTF-16 codec also is an autodetect feature (to detect the endianness), and I don't consider it a weird hack. Regards, Martin From regebro at gmail.com Mon Jan 11 18:27:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:27:01 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B5CF5.50806@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> Message-ID: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> On Mon, Jan 11, 2010 at 18:16, "Martin v. L?wis" wrote: >> But an autodetect feature is not a codec. Sure it should be reusable, >> but making it a codec seems to be ?a weird hack to me. > > Well, the existing UTF-16 codec also is an autodetect feature (to > detect the endianness), and I don't consider it a weird hack. So the BOM codec should raise a UnicodeDecodeError if there is no BOM? Because that's what it would have to do, in that case, because it can't fall back on anything, it has to handle and implement all encodings that have a BOM. And is it then actually very useful? You would have to do a try/except first with encoding='BOM' and then encoding=None to get the fallback to the standard. I must say that I find this whole thing pretty obvious. 'BOM' is not an encoding. Either there should be a method to get the encoding from the BOM, returning None of there isn't one, or open() should look at the BOM when you pass in encoding=None. Or both. That covers all usecases, is easy and obvious. Either open(file=foo, encoding=None) or open(file, encoding=encoding_from_bom(file)) I can't see that open(file, encoding='BOM') has any benefit over this, covers any extra usecase and is clearer in any way. Instead it adds something confusing: An encoding that isn't an encoding. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From python at mrabarnett.plus.com Mon Jan 11 18:35:35 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:35:35 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <4B4B6167.40606@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: >> UTF-8 might be a good choice > > No, fallback if there is no BOM should be the local settings, just as > fallback is today if you don't specify a codec. > I mean, what if you want to look for a BOM but fall back to something > else? How far will we go with encoding special information in the > codecs names? codec='BOM else UTF-16 else locale'? :-) > > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. It should either be default to look for a > BOM when codec=None, or a special parameter. If none of these are > desired, then we need a special function that takes a filename or file > handle, and looks for a BOM and returns the codec found or None. But > I find that much less natural and obvious than checking the BOM when codec=None. > Or pass a function that accepts a byte stream or the first few bytes and returns the encoding and any unused bytes (because the byte stream might not be seekable)? def guess_encoding(byte_stream): data = byte_stream.read(2) if data == b"\xFE\xFF": return "UTF-16BE", b"" return "UTF-8", data text_file = open(filename, encoding=guess_encoding) ... From python at mrabarnett.plus.com Mon Jan 11 18:46:30 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:46:30 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> Message-ID: <4B4B63F6.1020309@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" > wrote: >> I know you are just looking for a compromise, but this shouldn't be >> it: the PSF has deliberately stayed out of the actual Python >> engineering, so the release that Benjamin makes is not done by the >> PSF (but by Benjamin and his contributors :-). > > Hm. Yeah. That's right of course. I started with saying "official", > but then I thought "official according to who?". :) "Official" is whatever the BDFL says it is! :-) > So I changed it to mentioning PSF, but that doesn't work either. I > guess the current writing as as good as it gets, unless we change > "scheduled" to "expected" or something. > From martin at v.loewis.de Mon Jan 11 18:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:59:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> Message-ID: <4B4B66EC.7000203@v.loewis.de> > I must say that I find this whole thing pretty obvious. 'BOM' is not > an encoding. That I certainly agree with. > That covers all usecases, is easy and obvious. Either open(file=foo, > encoding=None) or open(file, encoding=encoding_from_bom(file)) > > I can't see that open(file, encoding='BOM') has any benefit over this, Well, it would have the advantage that Walter pointed out: you can implement it independent of the open() implementation, and even provide it in older versions of Python. Regards, Martin From regebro at gmail.com Mon Jan 11 18:59:52 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B63F6.1020309@mrabarnett.plus.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> Message-ID: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> On Mon, Jan 11, 2010 at 18:46, MRAB wrote: > "Official" is whatever the BDFL says it is! :-) Heh, right. So, it should say "Guido wants 2.7 to be the last main version of Python 2, so it probably will be. We promise to release bugfixes it for, like, ages". No need to be needlessly formal. :-D -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From olemis at gmail.com Mon Jan 11 19:58:01 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 13:58:01 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> [...] > I had similar issues too (please read below ;o) ... On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > About guessing the encoding, I experienced this issue while I was developing a Trac plugin. What I was doing is as follows : - I guessed the MIME type + charset encoding using Trac MIME API (it was a CSV file encoded using UTF-16) - I read the file using `open` - Then wrapped the file using `codecs.EncodedFile` - Then used `csv.reader` ... and still get the BOM in the first value of the first row in the CSV file. {{{ #!python >>> mimetype 'utf-16-le' >>> ef = EncodedFile(f, 'utf-8', mimetype) }}} IMO I think I am +1 for leaving `open` just like it is, and use module `codecs` to deal with encodings, but I am strongly -1 for returning the BOM while using `EncodedFile` (mainly because encoding is explicitly supplied in ;o) > --Guido > CMIIW anyway ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mal at egenix.com Mon Jan 11 21:44:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Jan 2010 21:44:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> Message-ID: <4B4B8DB2.1060102@egenix.com> Olemis Lang wrote: >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>> BOM should be "ignored". >>> > [...] >> > > I had similar issues too (please read below ;o) ... > > On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> > > About guessing the encoding, I experienced this issue while I was > developing a Trac plugin. What I was doing is as follows : > > - I guessed the MIME type + charset encoding using Trac MIME API (it > was a CSV file encoded using UTF-16) > - I read the file using `open` > - Then wrapped the file using `codecs.EncodedFile` > - Then used `csv.reader` > > ... and still get the BOM in the first value of the first row in the CSV file. You didn't say, but I presume that the charset guessing logic returned either 'utf-16-le' or 'utf-16-be' - those encodings don't remove the leading BOM. The 'utf-16' codec will remove the BOM. > {{{ > #!python > >>>> mimetype > 'utf-16-le' >>>> ef = EncodedFile(f, 'utf-8', mimetype) > }}} Same here: the UTF-8 codec will not remove the BOM, you have to use the 'utf-8-sig' codec for that. > IMO I think I am +1 for leaving `open` just like it is, and use module > `codecs` to deal with encodings, but I am strongly -1 for returning > the BOM while using `EncodedFile` (mainly because encoding is > explicitly supplied in ;o) Note that EncodedFile() doesn't do any fancy BOM detection or filtering. This is the job of the codecs. Also note that BOM removal is only valid at the beginning of a file. All subsequent BOM-bytes have to be read as-is (they map to a zero-width non-breaking space) - without removing them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 11 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Jan 11 22:12:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 07:12:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <4B4B9447.4060101@gmail.com> Benjamin Peterson wrote: > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors I would call it a documentation bug: by leaving out the __get__ you're telling Python to override *just* the setting of the attribute, while still using the normal lookup process for retrieving the attribute (i.e. allowing the attribute to be shadowed in the instance dictionary). Adding a "print('Descriptor Invoked')" to the __set__ method in your example: >>> x = X() >>> print(x.attr) <__main__.Descr object at 0x7f283b51ac50> >>> x.attr = 42 Descriptor invoked >>> print(x.attr) 42 >>> x.attr = 6*9 Descriptor invoked >>> print(x.attr) 54 Note that the behaviour here is still different from that of a data descriptor: with a data descriptor, once it gets shadowed in the instance dictionary, the descriptor is ignored *completely*. The only way to get the descriptor involved again is to eliminate the shadowing. The non-data descriptor with only __set__ is just choosing not to override the attribute lookup process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Mon Jan 11 22:29:38 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 16:29:38 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B8DB2.1060102@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> <4B4B8DB2.1060102@egenix.com> Message-ID: <24ea26601001111329i7f609aa1i9f5db7eb61f1fae7@mail.gmail.com> Probably one part of this is OT , but I think it could complement the discussion ;o) On Mon, Jan 11, 2010 at 3:44 PM, M.-A. Lemburg wrote: > Olemis Lang wrote: >>> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >>> wrote: >>>> Hi, >>>> >>>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>>> BOM should be "ignored". >>>> >> [...] >>> >> >> I had similar issues too (please read below ;o) ... >> >> On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >>> >> >> About guessing the encoding, I experienced this issue while I was >> developing a Trac plugin. What I was doing is as follows : >> >> - I guessed the MIME type + charset encoding using Trac MIME API (it >> was a CSV file encoded using UTF-16) >> - I read the file using `open` >> - Then wrapped the file using `codecs.EncodedFile` >> - Then used `csv.reader` >> >> ... and still get the BOM in the first value of the first row in the CSV file. > > You didn't say, but I presume that the charset guessing logic > returned either 'utf-16-le' or 'utf-16-be' Yes. In fact they return the full mimetype 'text/csv; charset=utf-16-le' ... ;o) > - those encodings don't > remove the leading BOM. ... and they should ? > The 'utf-16' codec will remove the BOM. > In this particular case there's nothing I can do, I have to process whatever charset is detected in the input ;o) >> {{{ >> #!python >> >>>>> mimetype >> 'utf-16-le' >>>>> ef = EncodedFile(f, 'utf-8', mimetype) >> }}} > > Same here: the UTF-8 codec will not remove the BOM, you have > to use the 'utf-8-sig' codec for that. > >> IMO I think I am +1 for leaving `open` just like it is, and use module >> `codecs` to deal with encodings, but I am strongly -1 for returning >> the BOM while using `EncodedFile` (mainly because encoding is >> explicitly supplied in ;o) > > Note that EncodedFile() doesn't do any fancy BOM detection or > filtering. ... directly. > This is the job of the codecs. > OK ... to come back to the scope of the subject, in the general case, I think that BOM (if any) should be handled by codecs, and therefore indirectly by EncodedFile . If that's a explicit way of working with encodings I'd prefer to use that wrapper explicitly in order to (encode | decode) the file and let the codec detect whether there's a BOM or not and ?adjust? `tell`, `read` and everything else in that wrapper (instead of `open`). > Also note that BOM removal is only valid at the beginning of > a file. All subsequent BOM-bytes have to be read as-is (they > map to a zero-width non-breaking space) - without removing them. > ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Test cases for custom query (i.e report 9) ... PASS (1.0.0) - http://simelo.hg.sourceforge.net/hgweb/simelo/trac-gviz/rev/d276011e7297 From martin at v.loewis.de Mon Jan 11 22:42:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:42:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: <4B4B9B55.1010709@v.loewis.de> > So the question we should be asking is: what's missing from Python > 2.7 to help with this transition? Wrt. the distribute feature you've noticed: Python 3.0 already supports that in distutils. So from day one of Python 3.x, you could have used that approach for porting Python code. I had been promoting it ever since, but it's taking up only slowly, partially because it has competition in the approaches "burn the bridges" (i.e. convert to 3.x once, and then have two code bases, or abandon 2.x), and "avoid 2to3" (i.e. try to write code that works in both 2.x and 3.x unmodified). > If we can't get it into 2.7, then > why, and would pushing it back to 2.8 help at all? I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: a) telling people that they have to move to 2.6 first actually hurts migration, instead of helping, because it implies to them that they have to drop old versions (e.g. 2.3.) - just because they had *always* dropped old versions before supporting new ones. b) IMO, people also don't gain much by first migrating to 2.6. In principle, it gives them the opportunity to get py3k warnings. However, I haven't heard a single positive report where these warnings have actually helped people in porting. Yours is the first report saying that you followed the official guideline, but you didn't state whether doing so actually helped (or whether you just ported to 2.6 because the guideline told you to). c) whatever 2.7 does (except perhaps for the warnings), it won't be useful to applications, because they couldn't use it, anyway: they need to support 2.4 and 2.5, and it won't have any of the gimmicks people come up with for 2.7. Adding gimmicks to 2.7 might actually hurt porting: people may have to special-case 2.7 because their work-arounds for older versions may break in 2.7 (e.g. testing whether a name is *not* defined, when it becomes defined in 2.7), plus it gives them an incentive to not port yet until they can drop support for anything before 2.7. Inherently, 2.8 can't improve on that. Regards, Martin From martin at v.loewis.de Mon Jan 11 22:44:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:44:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> Message-ID: <4B4B9BB8.3070505@v.loewis.de> > So, it should say "Guido wants 2.7 to be the last main version of > Python 2, so it probably will be. We promise to release bugfixes it > for, like, ages". > > No need to be needlessly formal. :-D That summarizes my understanding of what Guido said fairly well :-) +1 for putting it into the announcements. Regards, Martin From fuzzyman at voidspace.org.uk Mon Jan 11 23:11:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 11 Jan 2010 22:11:44 +0000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4B9447.4060101@gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> Message-ID: <4B4BA220.20701@voidspace.org.uk> On 11/01/2010 21:12, Nick Coghlan wrote: > Benjamin Peterson wrote: > > My question is: Is this a doc bug or a implementation bug? If the > >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors >> > [snip...] > > Note that the behaviour here is still different from that of a data > descriptor: with a data descriptor, once it gets shadowed in the > instance dictionary, the descriptor is ignored *completely*. The only > way to get the descriptor involved again is to eliminate the shadowing. > The non-data descriptor with only __set__ is just choosing not to > override the attribute lookup process. > > Does that mean we need a third class of descriptors that are neither data descriptors nor non-data descriptors? What should we call them: really-not-data-descriptors? All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Mon Jan 11 23:55:01 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Jan 2010 14:55:01 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9BB8.3070505@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: +1 from me. (With the caveat that "time will tell" is definitely the ultimate answer. Plans may change unexpectedly, even though we don't expect them to.) PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not helpful. But I trust he has ported a lot more code to 3.x than I have personally. Are there other experiences? --Guido On Mon, Jan 11, 2010 at 1:44 PM, "Martin v. L?wis" wrote: >> So, it should say "Guido wants 2.7 to be the last main version of >> Python 2, so it probably will be. We promise to release bugfixes it >> for, like, ages". >> >> No need to be needlessly formal. :-D > > That summarizes my understanding of what Guido said fairly well :-) > > +1 for putting it into the announcements. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Jan 12 00:16:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 00:16:47 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <4B4BB15F.5020807@v.loewis.de> Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. That's really because I haven't even attempted to use it. Instead, the software would fall flat on its own when running the test suite in 3.x. IOW, I don't need 2.6 to tell me what might break when I can see 3.x actually breaking. Regards, Martin From david.lyon at preisshare.net Tue Jan 12 00:22:42 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:22:42 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Hi Martin, > Of course, the less active fraction of Python contributors may not > notice, since they just chose to not contribute (which, of course, > is fine). However, asking me to work twice as much as I want to > on the project to keep two branches alive is just unfair. Totally true. Actually as an end-developer I'd say that python 2.x series from a programming perspective is quite good. It doesn't need the addition of a lot of new features imho. So for me, I think too much time spent there would be not yield great benefits. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Well, we all know your work is super quality. :-) That's not being contested. However, Quality can be measured different ways and it can be assessed in different ways. Quality itself is a subjective thing. The point I'm only making is that if a piece of software doesn't have "new" things added over time, then users can get a reverse impression of a lack of quality. We've all seen where 'internal' quality can increase and user perceptions can decrease. It could be things like improved graphics and things readily apparent to the user. At the moment, I would say that the "internal" quality of the python 2.x series is super high. "external" quality issues such as the packaging dilemma give the user the opposite quality experience. But people are working on that as best they can elsewhere. I'll leave it at that. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Python 3.x needs more carrots. >From an ordinary (perphaps ignorant) user perspective there is nothing. Yes, we know if we actually will start programming then we will like it more. But my wishes to Santa Claus would be allow the free flow of PEPs for Python 3 packaging. Even encourage it. As an end developer, here's what I'd like from Santa in 2010 to get me to swap to python 3: * get all the packages on pypi tested for python 3 * put a web based package manager in python 3. This would perhaps be based around PIP (keep many people happy) and would look much like the built in web-console that you get with a $200 router. * Incorporate SCM based end-user software installs by adding to python3-stdlib the SCM support packages (cvs,bzr,svn,hg). This would *really* help in the scientific community. * put a web interface on distutils also so that we don't have to use a command line interface. (I want a pic of a smiley girl to great me when I build something - "Are you ready to build now Honey?"). ok - I joke. But the point is made. So, ok, maybe these things aren't about 'code' quality. But rather user experience. Things like these do count also as "quality" via the technical term "perception of quality". If the PEP process is as unblocked as the documentation implies, implying that anybody can contribute to Python 3. Then there shouldn't be any issue. David From solipsis at pitrou.net Tue Jan 12 00:37:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 23:37:47 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon preisshare.net> writes: > > > This has nothing to do with pushing 3.x, but all with managing > > available manpower and still providing quality software. > > Python 3.x needs more carrots. As someone who experiences the difference almost every day, I can say 3.x definitely has enough carrots for me. The simple fact that I will be able to raise exceptions with non-ASCII error messages without having to workaround a hopelessly broken conversion/reporting logic is a huge improvement. And that's only a small part of the picture. Regards Antoine. From janssen at parc.com Tue Jan 12 00:47:55 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 11 Jan 2010 15:47:55 PST Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <17215.1263253675@parc.com> > David Lyon preisshare.net> writes: > > > > > This has nothing to do with pushing 3.x, but all with managing > > > available manpower and still providing quality software. > > > > Python 3.x needs more carrots. I'd be happy to move UpLib to Python 3, when the various packages that I need are also on Python 3. And that depresses me to think about. I need Medusa ReportLab PyLucene Email Vobject Mutagen Pyglet Hachoir Win32 I'm on the mailing lists for a lot of these, and the only one that I know is thinking about Python 3 is Email. I'd expect Win32 and PIL to also be thinking about it, but I haven't heard anything. Bill From david.lyon at preisshare.net Tue Jan 12 00:47:51 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:47:51 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <1220.218.214.45.58.1263253671.squirrel@syd-srv02.ezyreg.com> Antoine writes: > As someone who experiences the difference almost every day, I can say 3.x > definitely has enough carrots for me. The simple fact that I will be able > to > raise exceptions with non-ASCII error messages without having to > workaround a > hopelessly broken conversion/reporting logic is a huge improvement. And > that's > only a small part of the picture. In no way could I disagree with you. Ascii works fine for us here - but I guess we're lucky. Just keep pushing python 3 and have a nice day.. David From barry at python.org Tue Jan 12 01:11:28 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:11:28 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100111191128.39020d89@freewill> On Jan 11, 2010, at 10:42 PM, Martin v. L?wis wrote: >Wrt. the distribute feature you've noticed: Python 3.0 already supports >that in distutils. So from day one of Python 3.x, you could have used >that approach for porting Python code. I had been promoting it ever >since, but it's taking up only slowly, partially because it has >competition in the approaches "burn the bridges" (i.e. convert to 3.x >once, and then have two code bases, or abandon 2.x), and "avoid 2to3" >(i.e. try to write code that works in both 2.x and 3.x unmodified). Interesting. The only reason I never used it before is because I didn't know about it. Am I alone? Maybe I'm projecting my own preferences, but it seems to me that supporting both Python 2 and 3 from the same code base would be a very powerful way to enable a large amount of existing code to support Python 3. I'm certainly going to do this from now on with all the libraries I maintain. I don't have any cycles to write code twice and I can't abandon Python 2 yet. I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. As an example, the one library I've already ported used a metaclass. I don't see any way to specify that the metaclass should be used in a portable way. In Python 2.6 it's: class Foo: __metaclass__ = Meta and in Python 3 it's: class Foo(metaclass=Meta): 2to3 made that pain go away. >I've done a fair bit of 3.x porting, and I'm firmly convinced that >2.x can do nothing: > >a) telling people that they have to move to 2.6 first actually > hurts migration, instead of helping, because it implies to them > that they have to drop old versions (e.g. 2.3.) - just because > they had *always* dropped old versions before supporting new ones. Is it just an implication, or is it reality? I have yet to try to support older versions of Python and also support Python 3. (The one package I've done this with so far only supports Python 2.6.) >b) IMO, people also don't gain much by first migrating to 2.6. > In principle, it gives them the opportunity to get py3k warnings. > However, I haven't heard a single positive report where these > warnings have actually helped people in porting. Yours is the > first report saying that you followed the official guideline, > but you didn't state whether doing so actually helped (or whether > you just ported to 2.6 because the guideline told you to). Python 2.6 has other useful features, which I want to take advantage of, so getting py3k warnings is a bonus. Running 'python2.6 -3 setup.py test' over my code did in fact help clean up a couple of problems. Seems like a good first step when considering Python 3 support to me. >c) whatever 2.7 does (except perhaps for the warnings), it won't > be useful to applications, because they couldn't use it, anyway: > they need to support 2.4 and 2.5, and it won't have any of the > gimmicks people come up with for 2.7. Adding gimmicks to 2.7 > might actually hurt porting: people may have to special-case > 2.7 because their work-arounds for older versions may break in 2.7 > (e.g. testing whether a name is *not* defined, when it becomes > defined in 2.7), plus it gives them an incentive to not port > yet until they can drop support for anything before 2.7. > >Inherently, 2.8 can't improve on that. I'm not so sure. Yes, as a package maintainer there are older versions to think about, but time moves on for everyone (hopefully :). By the time 2.8 is released, what will be the minimum version of Python provided by most OS vendors (where the majority of Python users probably get their 'python')? I guess some people will have to support everything from Python 2.3 to 2.8 but you're talking supporting something like a spread of 7 years of Python versions. What other platform do you support for 7 years? Even Ubuntu long term support (LTS) releases only live for 5 years and even then, newer Pythons are often back ported. Or if not, then who cares? You're not going to use a newer version of a library on an LTS anyway. I'm not necessarily saying that justifies a 2.8 release. I'm just asking how we can make it easier for people to port to Python 3. The automatic 2to3 has already made it easier for me to port one library to Python 3 because I barely had to even think about it. Maybe that's enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 01:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:12:19 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <20100111191219.5fdd2542@freewill> On Jan 11, 2010, at 02:55 PM, Guido van Rossum wrote: >PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not >helpful. But I trust he has ported a lot more code to 3.x than I have >personally. Are there other experiences? I've only done one small library so far, but it was helpful to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From andrew at bemusement.org Tue Jan 12 01:07:26 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Tue, 12 Jan 2010 11:07:26 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100112000726.GO32351@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > I've done a fair bit of 3.x porting, and I'm firmly convinced that > 2.x can do nothing: [...] > Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. From solipsis at pitrou.net Tue Jan 12 01:28:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 00:28:26 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: Andrew Bennetts bemusement.org> writes: > > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. I don't think a 2.8 could make you much closer to 3.x. AFAICT, every possible way to ease transition to 3.x will already be in 2.7, or almost. But perhaps I'm lacking imagination. Regards Antoine. From brian.curtin at gmail.com Tue Jan 12 02:57:46 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 11 Jan 2010 19:57:46 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > -Brett > I think it would be interesting to see how people are using the tracker, or how they want to be using it. For example, there are currently over 1500 open issues with no stage set, some of which seemingly haven't been read by anyone at all. Would a properly set stage field save issues from falling into a black hole? Food for thought: according to the last tracker summary, there are over 1000 open issues with a patch, and issues stay open an average of 700 days (median 450). Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Jan 12 04:53:24 2010 From: jackdied at gmail.com (Jack Diederich) Date: Mon, 11 Jan 2010 22:53:24 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. [sidebar] 1) the metaclass fixer was a PITA to implement. 2) 95% of __metaclass__ definitions searchable via google code were of the "__metaclass__ = type" variety. The 2to3 patch exists only because of the few other uses. 3) 100% of the module level assignments in public projects were the "__metaclass__ = type" variety which is why there isn't a fixer for that. Also, a fixer would have been really, really ugly (munge every class definition in this module because there is a top level assignment). -Jack From benjamin at python.org Tue Jan 12 05:01:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 11 Jan 2010 22:01:40 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <1afaf6161001112001t5e7bab83t7d93f33fa11ce849@mail.gmail.com> 2010/1/11 Jack Diederich : > On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: >> As an example, the one library I've already ported used a metaclass. ?I don't >> see any way to specify that the metaclass should be used in a portable way. >> In Python 2.6 it's: >> >> class Foo: >> ? ?__metaclass__ = Meta >> >> and in Python 3 it's: >> >> class Foo(metaclass=Meta): >> >> 2to3 made that pain go away. > > [sidebar] > 1) the metaclass fixer was a PITA to implement. Does this make it any less useful, though? :) -- Regards, Benjamin From steven.bethard at gmail.com Tue Jan 12 06:57:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 11 Jan 2010 21:57:18 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 4:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. Actually there's a solution to this one too: FooBase = Meta('FooBase', (), {}) class Foo(FooBase): ... That should work in Python 2.X and 3.X. I've got argparse running on Python 2.3-3.1, and the changes were pretty easy. You can see them all in the revision here: http://code.google.com/p/argparse/source/detail?r=12 I have aspirations of putting all of the tricks I learned up up on the Wiki somewhere, but I just haven't had the time. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From regebro at gmail.com Tue Jan 12 07:30:10 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:30:10 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> On Mon, Jan 11, 2010 at 23:55, Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. But I trust he has ported a lot more code to 3.x than I have > personally. Are there other experiences? It doesn't warn for that many of the unportable problems, but I'm not sure it can warn for them either. It's certainly helpful, just not very much. :) I think the biggest help was added in 2.6.2, and that's warning for old integer division. It will also warn for modules that aren't supported anymore, if I remember correctly, and that's often helpful. But it won't warn for real tricky problems, like binary vs text in strings, and I don't see how it can either. And, I just realized, it doesn't warn for you using cmp or __cmp__ either, and 2to3 won't fix that, so it should actually warn for it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:49:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:49:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> On Tue, Jan 12, 2010 at 01:11, Barry Warsaw wrote: > Interesting. ?The only reason I never used it before is because I didn't know > about it. ?Am I alone? Maybe not, but the Distribute feature is there because IMO the distutils feature by itself isn't particularily useful. You need to write your own distutils extensions, in practice, and they are not trivial. Distribute has simply done it for you. :) > I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:53:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:53:20 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <319e029f1001112253x511f8109ja2300a022010ef99@mail.gmail.com> On Tue, Jan 12, 2010 at 01:07, Andrew Bennetts wrote: > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. ?I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. When 2.7 was discussed several people agreed that 2.7 should include as much 3.x stuff as possible to ease transition. That turned out to not be very much, so I'm not sure there is more. :) Unless of course, 2.8 starts including more of the refactored libraries, but that's a very minor issue in porting, so it won't help much. To really help, it needs to start implement things that break backwards compatibility. That would be weird. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Tue Jan 12 10:39:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:39:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4BA220.20701@voidspace.org.uk> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> <4B4BA220.20701@voidspace.org.uk> Message-ID: <4B4C435B.7080903@gmail.com> Michael Foord wrote: >> Note that the behaviour here is still different from that of a data >> descriptor: with a data descriptor, once it gets shadowed in the >> instance dictionary, the descriptor is ignored *completely*. The only >> way to get the descriptor involved again is to eliminate the shadowing. >> The non-data descriptor with only __set__ is just choosing not to >> override the attribute lookup process. >> >> > > Does that mean we need a third class of descriptors that are neither > data descriptors nor non-data descriptors? Not really - leaving out __get__ when defining __set__ just creates a data descriptor that happens to use the default attribute look-up process rather than defining a different one. (Note that I had the data/non-data terminology backwards in my previous message - I tend to think of the two kinds of descriptor as enforced and non-enforced respectively precisely because I have to think about it to remember that "functions are non-data descriptors and properties are data descriptors, hence non-data descriptors only define __get__ while data descriptors define __set__". I don't find the data/non-data terminology intuitive at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:44:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:44:18 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> Message-ID: <4B4C4472.10104@gmail.com> Lennart Regebro wrote: > And, I just realized, it doesn't warn for you using cmp or __cmp__ > either, and 2to3 won't fix that, so it should actually warn for it. I have a vague recollection that we tried to warn for that and ended up nixing the warning due to vast swarms of false alarms (or because you couldn't write correct 2.x code without triggering it). I'd be happy for someone to prove my recollection wrong though (i.e. by writing a patch that implements such warnings). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:48:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:48:57 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4C4589.70906@gmail.com> David Lyon wrote: >> This has nothing to do with pushing 3.x, but all with managing >> available manpower and still providing quality software. > > Python 3.x needs more carrots. As Guido has said a few times, the gains are far greater for *new* Python developers than they are for existing ones. Existing developers have to unlearn old habits and wait for libraries they already use to get ported. New developers can just get started with a much cleaner language. They don't have as rich a 3rd party ecosystem on Python 3 as they would on Python 2.x at this point in time, but unlike existing developers they also have the luxury of cherry-picking just those packages that already have Python 3 support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Jan 12 11:20:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:20:27 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit?: > > For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. I think most issues /have/ been read. It's just that for many of them, nobody is interested enough in or feels competent enough for fixing them. > Would a properly set stage field save issues from > falling into a black hole? I don't think this has anything to do with properly setting the stage field. We just have limited time and manpower. Perhaps one of our goals should be to reach out more to potential contributors. > Food for thought: according to the last tracker summary, there are over > 1000 open issues with a patch, and issues stay open an average of 700 > days (median 450). As for open issues with a patch, a "code review" button sending this patch to Rietveld (or any other similar tool) is something many of us have been hoping for :-) It would make reviewing easier, faster and more thorough (because you visualize things better than by looking at a diff). Regards Antoine. From solipsis at pitrou.net Tue Jan 12 11:36:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:36:49 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Tue, 12 Jan 2010 10:20:27 +0000, Antoine Pitrou a ?crit?: > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Speaking of which, Steve had something to say about it: http://holdenweb.blogspot.com/2009/11/comments-or-not-public-or- private.html cheers Antoine. From ncoghlan at gmail.com Tue Jan 12 13:10:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 22:10:14 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C66A6.2040601@gmail.com> Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : >> For example, there are currently over >> 1500 open issues with no stage set, some of which seemingly haven't been >> read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. There are actually a whole host of reasons issues can stagnate: - a feature request may seem reasonable (hence it doesn't get rejected outright), but the right API may not be clear (hence it doesn't get implemented in the near term) - a patch may be reviewed and found to have significant defects (or simply be overreaching the stated goal) but the original patch poster loses interest after meeting resistance in their ambition to fix something that is "obviously" broken - a problem may simply be hard to fix in a backwards compatible way (or even at all!) Ultimately it boils down to Antoine's two categories (lack of interest and lack of relevant expertise to make a final decision) but there are a lot of different ways to get into those two buckets. Because we aren't ruthless about pruning those kinds of issues out of the tracker they're the ones that are going to accumulate over time. I'd actually be interested to know what the issue stats are like when RFEs are excluded and when the search is limited to the items flagged as 'easy'. If easy bug fix issues are taking a long time to get closed than that would bother me a lot more than RFEs that sit on the tracker for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 12 13:14:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:14:47 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071447.675c8f24@freewill> On Jan 11, 2010, at 10:53 PM, Jack Diederich wrote: >3) 100% of the module level assignments in public projects were the >"__metaclass__ = type" variety which is why there isn't a fixer for >that. Also, a fixer would have been really, really ugly (munge every >class definition in this module because there is a top level >assignment). And almost certainly unnecessary. IME, those are all to easily make bare class definitions new style in Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 13:16:09 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:16:09 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071609.1dcfffa6@freewill> On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: >Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > >That should work in Python 2.X and 3.X. Ugly, but good call! :) >I've got argparse running on Python 2.3-3.1, and the changes were >pretty easy. You can see them all in the revision here: > > http://code.google.com/p/argparse/source/detail?r=12 > >I have aspirations of putting all of the tricks I learned up up on the >Wiki somewhere, but I just haven't had the time. The more resources we can provide people, both in code and in documentation, the better. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Tue Jan 12 13:29:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 12:29:12 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112071609.1dcfffa6@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> Message-ID: <4B4C6B18.7050008@voidspace.org.uk> On 12/01/2010 12:16, Barry Warsaw wrote: > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > >> Actually there's a solution to this one too: >> >> FooBase = Meta('FooBase', (), {}) >> class Foo(FooBase): >> ... >> >> That should work in Python 2.X and 3.X. >> > Ugly, but good call! :) > > There are all sorts of tricks. For example you can do exception handling that works with pre-2.6 syntax and 3.0 with a bare except and using sys.exc_info. It is horrible, but acceptable for short pieces of code (I have a couple of small modules that do this). I haven't yet tried converting larger code-bases to Python 3, but I think the workflow advocated by Martin is greatly preferable to the hacks and tricks needed to make the same codebase run under 2 & 3. Michael >> I've got argparse running on Python 2.3-3.1, and the changes were >> pretty easy. You can see them all in the revision here: >> >> http://code.google.com/p/argparse/source/detail?r=12 >> >> I have aspirations of putting all of the tricks I learned up up on the >> Wiki somewhere, but I just haven't had the time. >> > The more resources we can provide people, both in code and in documentation, > the better. > > Thanks! > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 12 14:09:50 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Jan 2010 14:09:50 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C749E.4040009@egenix.com> Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > * argparse (PEP 389) > * brief mention on still wanting to break out the stdlib from CPython > * an official policy on extension modules? (i.e. must have a pure Python > implementation which can mean a ctypes implementation unless given an > explicit waiver) > > That's everything from a stdlib perspective. I have half-baked ideas, but > nothing concrete enough to bring up unless people really want to discuss > stuff like how to potentially standardize module deprecation warnings, etc. > > In terms of me personally, I do plan to bring up at some point during the > summit these points which don't squarely fit during my time slot: > > * an official unofficial policy on how new proposed features should come to > us (i.e. working code to python-ideas with a shell of a PEP that includes > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > If there is something missing from the stdlib discussion that you think > should be brought up at the summit let me know. And if there is something > here you want to discuss before the summit let me know and I can start a > separate thread on it. Could you please put these things and the results up on the Python wiki ?! We're going to have a language summit at EuroPython this year as well and may want to continue/extend the discussion based on what you're doing at PyCon. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 12 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brian.curtin at gmail.com Tue Jan 12 15:33:06 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 12 Jan 2010 08:33:06 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Tue, Jan 12, 2010 at 04:20, Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : > > > > For example, there are currently over > > 1500 open issues with no stage set, some of which seemingly haven't been > > read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. > > > Would a properly set stage field save issues from > > falling into a black hole? > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Agreed, I didn't mean to place blame on the stage field, I just ran with how I view that field since it was mentioned. When I'm thinking in "code-writing developer mode" (tm), I'm more likely to go to issues which have a stage so I know what I'm going into -- what needs to be worked on. When I'm in project cleanup mode, I go by stageless issues -- what is necessary for this to begin or end work. Maybe others work differently...software projects take all kinds :-) Anyways, sorry for the off-topic. If this is a summit worthy discussion, cool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 17:12:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 11:12:28 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C66A6.2040601@gmail.com> References: <4B4C66A6.2040601@gmail.com> Message-ID: <20100112161228.300EA1D688D@kimball.webabinitio.net> On Tue, 12 Jan 2010 22:10:14 +1000, Nick Coghlan wrote: > There are actually a whole host of reasons issues can stagnate: > - a feature request may seem reasonable (hence it doesn't get rejected > outright), but the right API may not be clear (hence it doesn't get > implemented in the near term) > - a patch may be reviewed and found to have significant defects (or > simply be overreaching the stated goal) but the original patch poster > loses interest after meeting resistance in their ambition to fix > something that is "obviously" broken > - a problem may simply be hard to fix in a backwards compatible way (or > even at all!) I would add: - a patch is reviewed but the reviewer requests a second opinion before committing, which never arrives, and the original reviewer forgets about the issue. I have occasionally observed apparently moribund issues spring to life and get resolved when a triage person does something as simple as updating the releases to which an issue applies. > Ultimately it boils down to Antoine's two categories (lack of interest > and lack of relevant expertise to make a final decision) but there are a > lot of different ways to get into those two buckets. There's a third bucket here: lack of time. Sometimes there is interest and expertise, but no time. Worse, sometimes we end up waiting on the person with the expertise and interest but no time, when someone else with somewhat less expertise but more time should just go ahead and do the commit. (*That* one should be fixable.) > Because we aren't ruthless about pruning those kinds of issues out of > the tracker they're the ones that are going to accumulate over time. Another other category that hangs around in the tracker is items for which there simply isn't a consensus. I suppose those could be brought to python-dev for a final decision if someone wanted to tackle that task. And I don't think it is a lack of ruthlessness. I think it is the current community consensus that we leave these issues open in the tracker in case someone does come along and want to deal with them. (*) > I'd actually be interested to know what the issue stats are like when > RFEs are excluded and when the search is limited to the items flagged as > 'easy'. If easy bug fix issues are taking a long time to get closed than > that would bother me a lot more than RFEs that sit on the tracker for > years. I'd also be interested to know what the average and median lifetime of closed bug reports is. Not the metrics for open issues, but the metrics for closed ones. My theory is that we close a lot of bugs fairly promptly, but the ones in the above categories make the average age of *open* issues high. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls (*) For example, I'm currently going through all the open issues relating to the email package, and some of them are pretty darn old, yet still have useful content. From brett at python.org Tue Jan 12 18:40:13 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:40:13 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C749E.4040009@egenix.com> References: <4B4C749E.4040009@egenix.com> Message-ID: On Tue, Jan 12, 2010 at 05:09, M.-A. Lemburg wrote: > Brett Cannon wrote: > > Michael has given me the hg transition/stdlib time slot at the language > > summit this year. In regards to that I plan to lead a discussion on: > > > > * where we are at w/ the Hg transition (Dirkjan should be there and I did > a > > blog post on this topic recently: > > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > > * argparse (PEP 389) > > * brief mention on still wanting to break out the stdlib from CPython > > * an official policy on extension modules? (i.e. must have a pure Python > > implementation which can mean a ctypes implementation unless given an > > explicit waiver) > > > > That's everything from a stdlib perspective. I have half-baked ideas, but > > nothing concrete enough to bring up unless people really want to discuss > > stuff like how to potentially standardize module deprecation warnings, > etc. > > > > In terms of me personally, I do plan to bring up at some point during the > > summit these points which don't squarely fit during my time slot: > > > > * an official unofficial policy on how new proposed features should come > to > > us (i.e. working code to python-ideas with a shell of a PEP that includes > > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > > * any changes needed to the issue tracker to help with the workflow? > (stage > > field seems like a failed experiment and we now have several effective > > triage people who can help w/ guiding changes) > > > > If there is something missing from the stdlib discussion that you think > > should be brought up at the summit let me know. And if there is something > > here you want to discuss before the summit let me know and I can start a > > separate thread on it. > > Could you please put these things and the results up on the Python > wiki ?! > > We're going to have a language summit at EuroPython this year > as well and may want to continue/extend the discussion based > on what you're doing at PyCon. > > I expect there will be at least summary emails on what gets discussed. There is also a chance that it will be videotaped. -Brett > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 12 2010) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 18:47:50 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:47:50 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? >> (stage field seems like a failed experiment and we now have several >> effective triage people who can help w/ guiding changes) >> >> -Brett >> > I think it would be interesting to see how people are using the tracker, or > how they want to be using it. For example, there are currently over 1500 > open issues with no stage set, some of which seemingly haven't been read by > anyone at all. Would a properly set stage field save issues from falling > into a black hole? > > "Using" is the key word there. I know this thread went on about how bugs tend to end up being left open, but what I was proposing to talk about was whether there is any shift desired by the seasoned tracker users to help in their work. For instance, the Stage field was meant to help, but I don't think it is really getting used much, which makes it at best just a UI junk and at worst something to confuse new users. So I just wanted to discuss things as a group to see if we could streamline some things, add others, etc. At worst I will try to grab people like Mark D., R. David, Antoine, Georg, etc. at the summit (not sure exactly which the heavy bug closers will be there), take them aside, and flat-out ask what they need to make their jobs easier. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 19:29:06 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:29:06 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C6B18.7050008@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> <4B4C6B18.7050008@voidspace.org.uk> Message-ID: On Tue, Jan 12, 2010 at 04:29, Michael Foord wrote: > On 12/01/2010 12:16, Barry Warsaw wrote: > > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > > > Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > > That should work in Python 2.X and 3.X. > > > Ugly, but good call! :) > > > > > There are all sorts of tricks. For example you can do exception handling > that works with pre-2.6 syntax and 3.0 with a bare except and using > sys.exc_info. It is horrible, but acceptable for short pieces of code (I > have a couple of small modules that do this). > > I haven't yet tried converting larger code-bases to Python 3, but I think > the workflow advocated by Martin is greatly preferable to the hacks and > tricks needed to make the same codebase run under 2 & 3. > > In other words we need to pull together a HOWTO for Python source like the one for extension modules that Benjamin wrote and make it rather prominently linked from the Python 3 documentation index page. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 19:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 13:31:23 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? > >> (stage field seems like a failed experiment and we now have several > >> effective triage people who can help w/ guiding changes) > >> > > I think it would be interesting to see how people are using the tracker, or > > how they want to be using it. For example, there are currently over 1500 > > open issues with no stage set, some of which seemingly haven't been read by > > anyone at all. Would a properly set stage field save issues from falling > > into a black hole? > > > "Using" is the key word there. I know this thread went on about how bugs > tend to end up being left open, but what I was proposing to talk about was > whether there is any shift desired by the seasoned tracker users to help in > their work. For instance, the Stage field was meant to help, but I don't > think it is really getting used much, which makes it at best just a UI junk > and at worst something to confuse new users. So I just wanted to discuss > things as a group to see if we could streamline some things, add others, > etc. Well, I for one find the stage field useful. It reminds me at a glance where the issue is at in the workflow (and 'not set' is a valid place in the workflow that an issue sometimes stays in for a while for reason other than not having been triaged yet). I suspect that if we discussed it as a group (we who make the most changes on the tracker) we could improve it, and the interface in general. By the way, you could talk to people who aren't going to be at the summit on #python-dev; I think all the currently tracker-active people hang out there on a regular basis. I'll have to give some thought to what changes/improvements might be most useful, now that I've been doing this for a while. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Tue Jan 12 19:34:02 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:34:02 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> References: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> Message-ID: On Tue, Jan 12, 2010 at 10:31, R. David Murray wrote: > On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > > On Mon, Jan 11, 2010 at 17:57, Brian Curtin > wrote: > > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > > >> * any changes needed to the issue tracker to help with the workflow? > > >> (stage field seems like a failed experiment and we now have several > > >> effective triage people who can help w/ guiding changes) > > >> > > > I think it would be interesting to see how people are using the > tracker, or > > > how they want to be using it. For example, there are currently over > 1500 > > > open issues with no stage set, some of which seemingly haven't been > read by > > > anyone at all. Would a properly set stage field save issues from > falling > > > into a black hole? > > > > > "Using" is the key word there. I know this thread went on about how bugs > > tend to end up being left open, but what I was proposing to talk about > was > > whether there is any shift desired by the seasoned tracker users to help > in > > their work. For instance, the Stage field was meant to help, but I don't > > think it is really getting used much, which makes it at best just a UI > junk > > and at worst something to confuse new users. So I just wanted to discuss > > things as a group to see if we could streamline some things, add others, > > etc. > > Well, I for one find the stage field useful. It reminds me at a glance > where the issue is at in the workflow (and 'not set' is a valid place > in the workflow that an issue sometimes stays in for a while for reason > other than not having been triaged yet). I suspect that if we discussed > it as a group (we who make the most changes on the tracker) we could > improve it, and the interface in general. > > By the way, you could talk to people who aren't going to be at the > summit on #python-dev; I think all the currently tracker-active people > hang out there on a regular basis. > > Sounds reasonable. > I'll have to give some thought to what changes/improvements might be > most useful, now that I've been doing this for a while. How about everyone who is active on bugs.python.org give it some thought and we can try to have a discussion at some point in the relatively near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jan 12 21:58:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 06:58:49 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: <4B4C749E.4040009@egenix.com> Message-ID: <4B4CE289.6040709@gmail.com> Brett Cannon wrote: > I expect there will be at least summary emails on what gets discussed. > There is also a chance that it will be videotaped. The Wiki makes for a better summary archive though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jan 12 22:53:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:53:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <4B4CEF3D.8000107@v.loewis.de> >> a) telling people that they have to move to 2.6 first actually >> hurts migration, instead of helping, because it implies to them >> that they have to drop old versions (e.g. 2.3.) - just because >> they had *always* dropped old versions before supporting new ones. > > Is it just an implication, or is it reality? That's only the implication. However, this was precisely the dialogue when talking to Django. If you start with "start supporting 2.6", the immediate response, without listening further, was, "ok, wait until we drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). Then explain it to the individual you are talking to, wait for the next developer of the project step along, and see how he brings up the very same line of thinking (supporting new versions == dropping support for old versions). I think only part of that comes from the maintenance burden. The other part is that they *want* to drop support for old versions, so that they can eventually start using new features (e.g. generator expressions). So they welcome the requirement to support new versions as an excuse to drop old ones ("it is obvious that you have to drop 2.3 to support 3.2"). However, their users then won't let them drop old versions. > >> b) IMO, people also don't gain much by first migrating to 2.6. >> In principle, it gives them the opportunity to get py3k warnings. >> However, I haven't heard a single positive report where these >> warnings have actually helped people in porting. Yours is the >> first report saying that you followed the official guideline, >> but you didn't state whether doing so actually helped (or whether >> you just ported to 2.6 because the guideline told you to). > > Python 2.6 has other useful features, which I want to take advantage of I think you are a minority with that, being able to actually use the 2.6 features already. Many projects can't, as they have to support at least 2.4 still (so the with statement is right out). >> Inherently, 2.8 can't improve on that. > > I'm not so sure. Yes, as a package maintainer there are older versions to > think about, but time moves on for everyone (hopefully :). By the time 2.8 is > released, what will be the minimum version of Python provided by most OS > vendors (where the majority of Python users probably get their 'python')? "Current" Linux distributions will have 2.6 then. "Old" installations will have 2.4. > I > guess some people will have to support everything from Python 2.3 to 2.8 but > you're talking supporting something like a spread of 7 years of Python > versions. What other platform do you support for 7 years? I think 2.3 will really be gone by the time 2.8 might get released. Even with 2.7, you'd end up with a span of seven years, though. Python had been supporting Windows 95 for more than 7 years (I think rather 9 or 10), likewise Windows 3.1 before that. Python 2.7 will likely still support Windows 2000, which then will be ten years old. Solaris support will probably go back to Solaris 2.6, which will be 13 years old when Python 2.7 gets released. It's only the Linux (and OS X) releases that move so quickly. Regards, Martin From martin at v.loewis.de Tue Jan 12 22:56:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:56:55 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> Message-ID: <4B4CF027.4080007@v.loewis.de> > Maybe not, but the Distribute feature is there because IMO the > distutils feature by itself isn't particularily useful. You need to > write your own distutils extensions, in practice, and they are not > trivial. I wouldn't say that. My Django port works with bare distutils (as does Django itself), and it works fine. That distribute had to redo it is only because setuptools *replaces* the build_py command, as does the 2to3 support in distutils. So only if you have a different build_py already, you can't use what is in distutils. Regards, Martin From fuzzyman at voidspace.org.uk Tue Jan 12 23:02:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:02:34 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CEF3D.8000107@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> Message-ID: <4B4CF17A.1000503@voidspace.org.uk> On 12/01/2010 21:53, "Martin v. L?wis" wrote: >>> a) telling people that they have to move to 2.6 first actually >>> hurts migration, instead of helping, because it implies to them >>> that they have to drop old versions (e.g. 2.3.) - just because >>> they had *always* dropped old versions before supporting new ones. >>> >> Is it just an implication, or is it reality? >> > That's only the implication. However, this was precisely the dialogue > when talking to Django. If you start with "start supporting 2.6", the > immediate response, without listening further, was, "ok, wait until we > drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). > > Then explain it to the individual you are talking to, wait for the next > developer of the project step along, and see how he brings up the > very same line of thinking (supporting new versions == dropping support > for old versions). > > I think only part of that comes from the maintenance burden. The other > part is that they *want* to drop support for old versions, so that they > can eventually start using new features (e.g. generator expressions). > So they welcome the requirement to support new versions as an excuse > to drop old ones ("it is obvious that you have to drop 2.3 to support > 3.2"). However, their users then won't let them drop old versions. > > > I agree with Martin that the *perception* is that to use Python 2.6 to help you port to Python 3 you have to be willing to drop support for earlier versions of Python. >> >>> b) IMO, people also don't gain much by first migrating to 2.6. >>> In principle, it gives them the opportunity to get py3k warnings. >>> However, I haven't heard a single positive report where these >>> warnings have actually helped people in porting. Yours is the >>> first report saying that you followed the official guideline, >>> but you didn't state whether doing so actually helped (or whether >>> you just ported to 2.6 because the guideline told you to). >>> >> Python 2.6 has other useful features, which I want to take advantage of >> > I think you are a minority with that, being able to actually use the 2.6 > features already. Many projects can't, as they have to support at least > 2.4 still (so the with statement is right out). > > Well, the IronPython community has almost completely moved over to IronPython 2.6. :-) We tend to ship our Python runtime with our applications though. As it happens I'm now working with CPython on the server (Python 2.5) and IronPython in the browser where we are using 2.6. The new property feature is nice, as is having with without a __future__ import. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From regebro at gmail.com Tue Jan 12 23:03:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 23:03:41 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF027.4080007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> <4B4CF027.4080007@v.loewis.de> Message-ID: <319e029f1001121403x14ef7ec8x729fb80fa67feaef@mail.gmail.com> On Tue, Jan 12, 2010 at 22:56, "Martin v. L?wis" wrote: >> Maybe not, but the Distribute feature is there because IMO the >> distutils feature by itself isn't particularily useful. You need to >> write your own distutils extensions, in practice, and they are not >> trivial. > > I wouldn't say that. My Django port works with bare distutils (as > does Django itself), and it works fine. > > That distribute had to redo it is only because setuptools *replaces* > the build_py command, as does the 2to3 support in distutils. So only > if you have a different build_py already, you can't use what is in > distutils. Yeah, you are right, I misremembered. The actual additional feature is the support for the test command. Testing under Python 2 and 3 without it is annoying. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 12 23:04:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:04:37 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <4B4CF1F5.4050403@v.loewis.de> > [...] >> I've done a fair bit of 3.x porting, and I'm firmly convinced that >> 2.x can do nothing: > [...] >> Inherently, 2.8 can't improve on that. > > I agree that there are limitations like the ones you've listed, but I > disagree with your conclusion. Maybe you assume that it's just as hard > to move to 2.8 (using the py3k backports not available in say 2.5) as it > is to 3.x? Not at all, no. I'd rather expect that code that runs on 2.7 will run on 2.8 unmodified. > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? > I think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? > Then, when all of your dependencies (or viable alternatives to those > dependencies) are available for 3.x, you'll have an easier transition if > you can start from a 2.x with fewer differences in features. But you won't *have* fewer differences. Just because your code runs on 2.8 doesn't mean it will stop running on 2.3 (if you have a need for that). This doesn't get you any closer - you can't use any of the 2.8 features as long as you have to support older versions of Python. > Fundamentally the more 2.x can converge on 3.x, the easier it will be > for users to make the leap, because it will be a smaller leap. No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. > The > longer the 2.x series lives, the more these newer 2.x versions like 2.7 > and maybe even 2.8 will be available on common platforms for people to > depend upon as minimum versions, which means that as time goes by they > can depend on a version that's closer to 3.x. No, that's incorrect. Suppose 2.7 is the last 2.x release, to be released in 2010. Then, in 2015, it may be that everybody has migrated to 2.7, which then is a common platform. If you release 2.8 in 2012, then, in 2015, people will be split between 2.7 and 2.8, and so there won't be a common platform before 2017. So stopping 2.x development *earlier* will also give us a common platform earlier. Regareds, Martin From martin at v.loewis.de Tue Jan 12 23:07:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:07:02 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4CF286.5040002@v.loewis.de> > I think it would be interesting to see how people are using the tracker, > or how they want to be using it. For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. Would a properly set stage field save issues from > falling into a black hole? I personally don't think this would be the case. What would help is people who actually *work* on the issues, rather than merely reading them. However, this being open source, there is no way to force it (unless you create an incentive, such as the five-for-one deal). Regards, Martin From python at mrabarnett.plus.com Tue Jan 12 23:10:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jan 2010 22:10:28 +0000 Subject: [Python-Dev] regex module Message-ID: <4B4CF354.2050603@mrabarnett.plus.com> Hi all, I'm back on the regex module after doing other things and I'd like your opinion on a number of matters: Firstly, the current re module has a bug whereby it doesn't split on zero-width matches. The BDFL has said that this behaviour should be retained by default in case any existing software depends on it. My question is: should my regex module still do this for Python 3? Speaking personally, I'd like it to behave correctly, and Python 3 is the version where backwards-compatibility is allowed to be broken. Secondly, Python 2 is reaching the end of the line and Python 3 is the future. Should I still release a version that works with Python 2? I'm thinking that it could be confusing if new regex module did zero-width splits correctly in Python 3 but not in Python 2. And also, should I release it only for Python 3 as a 'carrot'? Finally, the module allows some extra backslash escapes, eg \g, in the pattern. Should it treat ill-formed escapes, eg \g, as it would have treated them in the re module? Thanks From michael at voidspace.org.uk Tue Jan 12 23:46:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:46:49 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 Message-ID: <4B4CFBD9.1090009@voidspace.org.uk> I presume the email below is about the Windows binary. Does the AMD64 release work on intel 64bit and can we make the wording clearer on the download page? The current description is " Windows AMD64 binary". All the best, Michael -------- Original Message -------- Subject: Download Page - AMD64 Date: Fri, 11 Dec 2009 04:03:16 -0600 From: Thomas Brownback To: webmaster at python.org Should I use the AMD64 version of Python on an Intel 64 chip? I know those 64-bit implementations are very similar, but are they similar enough that your AMD64 will work on Intel? If so, perhaps a note should be placed on that page to help avoid confusion. Explicit being better than implicit and all. Thanks for your time, Thomas -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at bemusement.org Tue Jan 12 23:49:56 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Wed, 13 Jan 2010 09:49:56 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100112224956.GC28576@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > But a hypothetical 2.8 would also give people a way to move closer to > > py3k without giving up on using all their 2.x-only dependencies. > > How so? If they use anything that is new in 2.8, they *will* need to > drop support for anything before it, no??? > > > I think it's much more likely that libraries like Twisted can support 2.8 > > in the near future than 3.x. > > Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does > that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. From tjreedy at udel.edu Tue Jan 12 23:51:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 17:51:31 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: On 1/12/2010 5:04 PM, "Martin v. L?wis" wrote: > But you won't *have* fewer differences. Just because your code runs > on 2.8 doesn't mean it will stop running on 2.3 (if you have a need > for that). This doesn't get you any closer - you can't use any of > the 2.8 features as long as you have to support older versions of > Python. > >> Fundamentally the more 2.x can converge on 3.x, the easier it will be >> for users to make the leap, because it will be a smaller leap. > > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. > >> The >> longer the 2.x series lives, the more these newer 2.x versions like 2.7 >> and maybe even 2.8 will be available on common platforms for people to >> depend upon as minimum versions, which means that as time goes by they >> can depend on a version that's closer to 3.x. > > No, that's incorrect. Suppose 2.7 is the last 2.x release, to be > released in 2010. Then, in 2015, it may be that everybody has migrated > to 2.7, which then is a common platform. > > If you release 2.8 in 2012, then, in 2015, people will be split between > 2.7 and 2.8, and so there won't be a common platform before 2017. Just like people today may need to work with both 2.5 and 2.6, or privately backport 2.6 bugfixes to 2.5. > So stopping 2.x development *earlier* will also give us a common > platform earlier. With years of bug fixes and hence high quality. From tjreedy at udel.edu Wed Jan 13 00:05:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 18:05:12 -0500 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On 1/12/2010 5:10 PM, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. Are you writing a new module with a new name? If so, do you expect it to replace or augment re? (This is the same question as for optparse vs. argparse, which I understand to not yet be decided.) > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, 2.7 stdlib is already out. A new engine should get some community testing before going in the stdlib. Even 3.2 beta is not that far off (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? What does re do with analogous cases? Terry Jan Reedy From david.lyon at preisshare.net Wed Jan 13 00:20:54 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 13 Jan 2010 10:20:54 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C4589.70906@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> <4B4C4589.70906@gmail.com> Message-ID: <1333.218.214.45.58.1263338454.squirrel@syd-srv02.ezyreg.com> > Nick wrote: >>> This has nothing to do with pushing 3.x, but all with managing >>> available manpower and still providing quality software. >> >> Python 3.x needs more carrots. > > As Guido has said a few times, the gains are far greater for *new* > Python developers than they are for existing ones. Well both you and Guido are most likely 100% correct. :-) > They don't have as rich a 3rd party ecosystem > on Python 3 as they would on Python 2.x at this point in time, but > unlike existing developers they also have the luxury of cherry-picking > just those packages that already have Python 3 support. Most likely. I wouldn't want to say anything to discourage people from going to python 3. In other languages, I have much experience of making the jump to 'a whole new world'. It's unsettling at first, but after that, you suddenly find the new model has a better turbo than the last and you're at the next corner faster than you can think. So it's all good. But I still maintain Python 3.0 needs more carrots. For example, if mercurial or any other cool lib gets added to python 3 (and I can name a few that should go in python 3) then they should be added to python 3 and not to python 2.x They would serve as good carrots. Make fresh the python 3 stdlib and preserve the python 2.x stdlib. I really think we are somewhat short on resources to do what Guido has asked about bringing python up to CPAN level with respect to packages. We're starting a long way back with horses and swords and trying to catch a well fed and greased modern machine.. I'm sure we can get a modern package testbot operational for python. But I wish those who were complaining about the packaging problem so much could throw some money at the problem instead of moaning about it. As is done on other open-source projects. Some organisation would be beneficial. Finding funds so that a small group of people could work on the problem would be a great boost to forward progress. I just think python packaging needs six months of that to repair the years and years of neglect and stagnation. Even if it is only beer and bus fare money. It just needs a temporary shot of adrenalin in the arm.. so to speak.. Regards David From martin at v.loewis.de Wed Jan 13 00:28:14 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 00:28:14 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D058E.4030404@v.loewis.de> > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? "intel 64bit" is as clear as mud. It could mean the "Intel 64" architecture, or it could mean the "IA-64" architecture, both are 64-bit architectures with processors manufactured by Intel. The former is officially called AMD64 (not by Intel, but officially), and our AMD64 binaries run on such processors. The latter is also called "Itanium", and we stopped producing binaries for that architecture a while ago; the AMD64 binaries will *not* run on it. The wording could probably be changed to match the reader's expectation more, most likely, it would also get more incorrect in the process. To make it correct and explicit, an entire paragraph educating users about 64-bit architectures and that there are many of them and that they are mutually incompatible might be required. Most likely, Thomas' processor implements the AMD64 architecture, even though it is manufactured by Intel. A short sentence that he would probably understand (given that he used "Intel 64", not "64bit") is: """The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).""" Regards, Martin From martin at v.loewis.de Wed Jan 13 00:30:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 00:30:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112224956.GC28576@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> <20100112224956.GC28576@steerpike.home.puzzling.org> Message-ID: <4B4D0613.1010503@v.loewis.de> > I'm not talking about Twisted moving to 3.x (FWIW, I think the only > movement there so far is some patches for some -3 warnings). The > situation I'm describing is a project X that: > > (a) has 2.x-only dependencies, and > (b) would like to be as close as possible to 3.x (because they like > the new features and/or want to be as ready as possible to jump > when (a) is fixed). This assumes that jumping to 3.x is easier if you are closer to it. Please trust me that this is not the case. >> No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they >> likely won't. > > But this is my point. I think they would as an intermediate step to > jumping to 3.x (which also requires dropping 2.5, after all!), if for > some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. No, and no. No: it's not an intermediate step, and no, supporting 3.x does *not* require dropping 2.5. Regards, Martin From fuzzyman at voidspace.org.uk Wed Jan 13 00:40:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:40:35 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D058E.4030404@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> Message-ID: <4B4D0873.5070006@voidspace.org.uk> On 12/01/2010 23:28, "Martin v. L?wis" wrote: > [snip...] > """The binaries for AMD64 will also work on processors that implement > the Intel 64 architecture (formerly EM64T), i.e. the architecture that > Microsoft calls x64, and AMD called x86-64 before calling it AMD64. > They will not work on Intel Itanium Processors (formerly IA-64).""" > > To those not intimately aware of the history and present of 64 bit architecture this sentence would be a great addition - thanks. I'm adding it now as a footnote. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From lists at cheimes.de Wed Jan 13 00:41:04 2010 From: lists at cheimes.de (Christian Heimes) Date: Wed, 13 Jan 2010 00:41:04 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D0890.2030505@cheimes.de> Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". The installer works on all AMD64 compatible Intel CPUs. *scnr* As you most likely know all modern Intel 64bit CPUs are based on AMD's x86-64 design. Only the Itanium family is based on the other Intel 64bit design: IA-64. The name AMD64 was chosen because most Linux and BSD distributions call it so. The name 'AMD64' has caused confusion in the past because users thought that the installer works only on AMD CPUs. How about: * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / X86-64 binary -- does not include source) instead of: * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does not include source) ? Christia From fuzzyman at voidspace.org.uk Wed Jan 13 00:55:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:55:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0873.5070006@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> <4B4D0873.5070006@voidspace.org.uk> Message-ID: <4B4D0BD4.5050109@voidspace.org.uk> On 12/01/2010 23:40, Michael Foord wrote: > On 12/01/2010 23:28, "Martin v. L?wis" wrote: >> [snip...] >> """The binaries for AMD64 will also work on processors that implement >> the Intel 64 architecture (formerly EM64T), i.e. the architecture that >> Microsoft calls x64, and AMD called x86-64 before calling it AMD64. >> They will not work on Intel Itanium Processors (formerly IA-64).""" >> > > To those not intimately aware of the history and present of 64 bit > architecture this sentence would be a great addition - thanks. I'm > adding it now as a footnote. > I can add it now to the main download page and existing release pages - but are there changes needed to any scripts to change pages created for *new* releases? All the best, Michael Foord > All the best, > > Michael Foord > >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From python at mrabarnett.plus.com Wed Jan 13 01:22:01 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 00:22:01 +0000 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D1229.70708@mrabarnett.plus.com> Terry Reedy wrote: > On 1/12/2010 5:10 PM, MRAB wrote: >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. > > Are you writing a new module with a new name? If so, do you expect it to > replace or augment re? (This is the same question as for optparse vs. > argparse, which I understand to not yet be decided.) > It's a module called 'regex'. It can be used in place of 're' by using "import regex as re", except for differences such as "\g" being a legal group reference in pattern strings. >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? > > 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, > 2.7 stdlib is already out. A new engine should get some community > testing before going in the stdlib. Even 3.2 beta is not that far off > (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? > > What does re do with analogous cases? > The 're' module treats r"\g" as "g"; both 're' and 'regex' treat, say, r"\q" as "q". The closest analogue to what I'm asking about is that re treats the ill-formed repeat r"x{1," as a literal, which sort of suggests that r"\g" should be treated as "g", but r"\g" is now a group reference (re would treat that as "g". Does that sound reasonable? From fuzzyman at voidspace.org.uk Wed Jan 13 01:50:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 00:50:39 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D18DF.6070502@voidspace.org.uk> On 12/01/2010 23:41, Christian Heimes wrote: > Michael Foord wrote: > >> I presume the email below is about the Windows binary. Does the AMD64 >> release work on intel 64bit and can we make the wording clearer on the >> download page? >> >> The current description is " Windows AMD64 binary". >> > The installer works on all AMD64 compatible Intel CPUs. *scnr* > > As you most likely know all modern Intel 64bit CPUs are based on AMD's > x86-64 design. Only the Itanium family is based on the other Intel 64bit > design: IA-64. The name AMD64 was chosen because most Linux and BSD > distributions call it so. The name 'AMD64' has caused confusion in the > past because users thought that the installer works only on AMD CPUs. > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) > Right - I've made that change for the Python 2.6, 2.7, 3.0 and 3.1 download pages with a footnote. Prior to that we were offering ia64 *and* amd64 releases so I've left those unchanged. Thanks, Michael > ? > > Christia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From exarkun at twistedmatrix.com Wed Jan 13 02:40:03 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 13 Jan 2010 01:40:03 -0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100113014003.1898.1305834328.divmod.xquotient.219@localhost.localdomain> On 12 Jan, 10:04 pm, martin at v.loewis.de wrote: >>[...] >>>I've done a fair bit of 3.x porting, and I'm firmly convinced that >>>2.x can do nothing: >>[...] >>>Inherently, 2.8 can't improve on that. >> >>I agree that there are limitations like the ones you've listed, but I >>disagree with your conclusion. Maybe you assume that it's just as >>hard >>to move to 2.8 (using the py3k backports not available in say 2.5) as >>it >>is to 3.x? > >Not at all, no. I'd rather expect that code that runs on 2.7 will run >on 2.8 unmodified. >>But a hypothetical 2.8 would also give people a way to move closer to >>py3k without giving up on using all their 2.x-only dependencies. > >How so? If they use anything that is new in 2.8, they *will* need to >drop support for anything before it, no??? >>I think it's much more likely that libraries like Twisted can support >>2.8 >>in the near future than 3.x. > >Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does >that help Twisted in moving to 3.2? I'm not reading this thread carefully enough to make any arguments on either side, but I can contribute a fact. Twisted very likely does not support 2.8 today. I base this on the fact that Twisted does not support 2.7 today, and I expect 2.8 will be more like 2.7 than it will be like 2.3 - 2.6 (which Twisted does support). When I say "support" here, I mean "all of the Twisted unit tests pass on it". Jean-Paul From tseaver at palladion.com Wed Jan 13 03:57:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jan 2010 21:57:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4D36AA.7020607@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karen Tracey wrote: > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> >> > Well yay. Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference. That's > down from 104 failures and 40 errors with 2.7 alpha 1. > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Just to add another "success" data point: Zope2's trunk, as well as the 2.12 release, passes all tests (2535 on the trunk) and brings up the appserver just fine under 2.7a2. There is an obnoxious deprecation warning out of the distutils: DeprecationWarning: 'compiler' specifies the compiler type in build_ext. If you want to get the compiler object itself, use 'compiler_obj' which is likely a simple one-line fix, if I only knew what the hell it was whining about. ;) The warning is extra obnoxious because it doesn't tell me what in *my* code triggers the warning (it (needs a 'stacklevel' argument). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktNNqQACgkQ+gerLs4ltQ7d5gCcCGKVjyOlxnrAln0UnRibS7kd lNIAoIs1RlSGMtJWaY11BqptfDmQvR87 =mIOO -----END PGP SIGNATURE----- From python at mrabarnett.plus.com Wed Jan 13 04:09:34 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 03:09:34 +0000 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D396E.4050500@mrabarnett.plus.com> MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > I've just noticed something odd about the re module: the sub() method doesn't take 'pos' or 'endpos' arguments. search() does; match() does; findall() does(); finditer() does; but sub() doesn't. Maybe there has never been a demand for it. (Nor split(), for that matter.) From brett at python.org Wed Jan 13 04:58:11 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 19:58:11 -0800 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On Tue, Jan 12, 2010 at 14:10, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > If it is a separate module under a different name it can do the proper thing. People will just need to be aware of the difference when they import the module. > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > That's totally up to you. There is practically no chance of it getting into the 2.x under the stdlib at this point since 2.7b1 is coming up and this module has not been out in the wild for a year (to my knowledge). If you want to support 2.x that's fine and I am sure users would appreciate it, but it isn't necessary to get into the Python 3 stdlib. > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > > If you want to minimize the differences then it should probably match. As I said, since it is a different name to import under it can deviate where reasonable, just make sure to clearly document the deviations. -Brett > Thanks > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jan 13 07:11:49 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 07:11:49 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D6425.3060307@v.loewis.de> > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) -1. AMD doesn't want us to use the term x86-64 anymore, but wants us to use AMD64 instead. I think we should comply - they invented the architecture, so they have the right to give it a name. Neither Microsoft nor Intel have such a right. Regards, Martin From sridharr at activestate.com Wed Jan 13 07:47:38 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 12 Jan 2010 22:47:38 -0800 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D6C8A.7070307@activestate.com> On 1/12/2010 2:46 PM, Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". FWIW, we simply use (64-bit, x64). Platform Download ============================================== Windows (x86) Windows Installer (MSI) Windows (64-bit, x64) Windows Installer (MSI) https://www.activestate.com/activepython/downloads/ -srid From solipsis at pitrou.net Wed Jan 13 08:08:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Jan 2010 07:08:55 +0000 (UTC) Subject: [Python-Dev] Fwd: Download Page - AMD64 References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) +1. I don't care about trademarks or official names, we should call it whatever is obvious for our users. As for Itanium, it is practically dead, I think there is little chance of people mixing it up with x86-64. cheers Antoine. From michael at voidspace.org.uk Wed Jan 13 10:32:00 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 09:32:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4D9310.6060907@voidspace.org.uk> On 13/01/2010 06:11, "Martin v. L?wis" wrote: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) >> >> instead of: >> >> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >> not include source) >> > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. > I think we should use whatever is most informative and least confusing to our users - we owe our allegiance to them and not to a processor vendor. Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Wed Jan 13 11:30:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:30:50 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF17A.1000503@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> Message-ID: <4B4DA0DA.7070007@gmail.com> Michael Foord wrote: > I agree with Martin that the *perception* is that to use Python 2.6 to > help you port to Python 3 you have to be willing to drop support for > earlier versions of Python. Note that increased 3.x compatibility in the most recent 2.x release will always help in two scenarios: 1. New projects that want to use 2.x only libraries but want to be ready for the Py3k transition in their own code (e.g. new 2.7 features like set literals, dict and set comprehensions and the view* dict methods can be translated far more cleanly by 2to3 than the closest comparable 2.6 code). 2. Similarly new projects that use a 3.x trunk can be more readily translated to a 2.7 version with 3to2, whereas some constructs may be difficult to recreate in earlier Python versions. I would expect this to be significantly less common then the first scenario though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:38:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:38:31 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4DA2A7.2080408@gmail.com> Antoine Pitrou wrote: > Christian Heimes cheimes.de> writes: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) > > +1. I don't care about trademarks or official names, we should call it whatever > is obvious for our users. Until this discussion, I didn't even know the architecture had any name other than x86-64. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:39:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:39:40 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4D36AA.7020607@palladion.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> Message-ID: <4B4DA2EC.3050908@gmail.com> Tres Seaver wrote: > There is an obnoxious deprecation warning out of the distutils: > > DeprecationWarning: 'compiler' specifies the compiler type in > build_ext. If you want to get the compiler object itself, use > 'compiler_obj' > > which is likely a simple one-line fix, if I only knew what the hell it > was whining about. ;) The warning is extra obnoxious because it doesn't > tell me what in *my* code triggers the warning (it (needs a 'stacklevel' > argument). Could you kick a tracker issues in Tarek's direction for that one? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Wed Jan 13 13:24:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 13 Jan 2010 12:24:09 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Brett Cannon python.org> writes: > If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. I'm sorry I won't be able to come to the language summit, but I would like if possible to expedite a pronouncement on PEP 391 (configuring logging using dictionaries). I believe I addressed all the comments made on the discussion threads mentioned in the PEP and so I'm not sure what more I need to do to get a pronouncement. I guess the stdlib slot gives an opportunity for people to air their views and so I'd be grateful if you added it to the agenda. Thanks & regards, Vinay Sajip From murman at gmail.com Wed Jan 13 14:35:51 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 07:35:51 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. I see and agree with the motivation behind your point, but it's just as reasonable to mimic the name Microsoft uses: the release is for Windows rather than the processor. On MSDN Microsoft uses parentheticals x86, ia64 and x64; this would suggest a name like Python 2.6.4 installer for Windows (x64). Unfortunately this usage doesn't seem to be reflected in consumer-oriented product pages, so I'm uncertain how clear it is for those downloading Python. -- Michael Urman From ncoghlan at gmail.com Wed Jan 13 15:04:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 00:04:28 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4DD2EC.3030709@gmail.com> Michael Urman wrote: > On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. > > I see and agree with the motivation behind your point, but it's just > as reasonable to mimic the name Microsoft uses: the release is for > Windows rather than the processor. On MSDN Microsoft uses > parentheticals x86, ia64 and x64; this would suggest a name like > Python 2.6.4 installer for Windows (x64). Unfortunately this usage > doesn't seem to be reflected in consumer-oriented product pages, so > I'm uncertain how clear it is for those downloading Python. As Windows doesn't run on non-x86 architectures, their naming is generally just Windows (32 bit) and Windows (64 bit). Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. In this case, making it clear that the 64-bit Windows binaries work for both Intel and AMD chips is more important than using only the official architecture name. Cheers, Nick. P.S. This discussion made me realise that out of habit I had dropped the 32-bit version of Python on my new 64-bit Windows box... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Wed Jan 13 17:49:55 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 13 Jan 2010 11:49:55 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA2EC.3050908@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> <4B4DA2EC.3050908@gmail.com> Message-ID: <4B4DF9B3.4030403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Tres Seaver wrote: >> There is an obnoxious deprecation warning out of the distutils: >> >> DeprecationWarning: 'compiler' specifies the compiler type in >> build_ext. If you want to get the compiler object itself, use >> 'compiler_obj' >> >> which is likely a simple one-line fix, if I only knew what the hell it >> was whining about. ;) The warning is extra obnoxious because it doesn't >> tell me what in *my* code triggers the warning (it (needs a 'stacklevel' >> argument). > > Could you kick a tracker issues in Tarek's direction for that one? Done: http://bugs.python.org/issue7694 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktN+bMACgkQ+gerLs4ltQ6s4gCdENhtVQWzoH449BCDz8SNx/4s DEoAn19LMcMs3b6ctdNF8K2hYd6d+BSZ =TWit -----END PGP SIGNATURE----- From rdmurray at bitdance.com Wed Jan 13 18:24:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Jan 2010 12:24:42 -0500 Subject: [Python-Dev] PYTHON3PATH Message-ID: <20100113172442.4A7771FA679@kimball.webabinitio.net> Please review issue 2375 [1], which is an enhancement request to add a PYTHON3PATH environment variable. Because we have elected to have both a python and a python3 command, I think this is an issue worth thinking about carefully to make sure we are serving the Python user community and easing the transition to python3. It could be that "use virtualenv" is the best answer, but I feel we should think about it carefully to make sure that is really true. [1] http://bugs.python.org/issue2375 -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From guido at python.org Wed Jan 13 18:31:39 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:31:39 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: Somehow the bug site doesn't load for me right now, but I'm -1 on this. There are maybe a dozen PYTHON... variables -- we really shouldn't try to add PYTHON3 variants for all of them. Specifically for PYTHONPATH, I feel that its use is always a short-time or localized hack, not something you set in your .profile nd forget about it (forgetting about it inparticular often leads to unnecessary debugging pain later). The better approaches are based on site-packages, wrapper scripts, virtualenv, etc. --Guido On Wed, Jan 13, 2010 at 9:24 AM, R. David Murray wrote: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. ?Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. ?It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 > > -- > R. David Murray ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dirkjan at ochtman.nl Wed Jan 13 18:38:26 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 13 Jan 2010 18:38:26 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 18:29, Benjamin Peterson wrote: > Please consider trying Python 2.7 with your code and reporting any bugs you may I ran the Mercurial test suite on current trunk, and it worked alright. There was a small issue with the fact that mimetypes got fooled by the lack of ImportError from _winreg (which I solved by blacklisting _winreg in demandimport), and one test fails likely because of a change in MIME handling. Will look into that. So the state of trunk looks rather solid from where I sit. Cheers, Dirkjan From ralf at brainbot.com Wed Jan 13 18:40:04 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:40:04 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> (R. David Murray's message of "Wed, 13 Jan 2010 12:24:42 -0500") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87r5ptdhu3.fsf@muni.brainbot.com> "R. David Murray" writes: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 The first thing I got while trying to run a python3 prompt few days ago, was an error. python3 tried to read my $PYTHONSTARTUP file, which used print statements. people will have to run both python 2 and python 3 code at the same time. Using different environment variables will make this easier. - Ralf From ralf at brainbot.com Wed Jan 13 18:55:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:55:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Guido van Rossum's message of "Wed, 13 Jan 2010 09:31:39 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87k4vldh4c.fsf@muni.brainbot.com> Guido van Rossum writes: > Somehow the bug site doesn't load for me right now, but I'm -1 on > this. There are maybe a dozen PYTHON... variables -- we really > shouldn't try to add PYTHON3 variants for all of them. > > Specifically for PYTHONPATH, I feel that its use is always a > short-time or localized hack, not something you set in your .profile > nd forget about it (forgetting about it inparticular often leads to > unnecessary debugging pain later). The better approaches are based on > site-packages, wrapper scripts, virtualenv, etc. then at least remove them completely from python3, so one can use them for his python 2 installation. Otherwise it will stump on some innocent python 3 program (and cause unnecessary debugging pain). From steven.bethard at gmail.com Wed Jan 13 18:57:29 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 13 Jan 2010 09:57:29 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: On Wed, Jan 13, 2010 at 9:40 AM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. ?Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. ?It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. How complicated is your PYTHONSTARTUP file? My suspicion is that you could easily write it to work for both Python 2.X and 3.X. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ssteinerx at gmail.com Wed Jan 13 18:57:42 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 13 Jan 2010 12:57:42 -0500 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> On Jan 13, 2010, at 12:40 PM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. Python 2 will continue to run as it has, and better solutions will emerge for Python 3 development. Beats the heck out of making a whole duplicate set for PYTHON3BLAHBLAH, yuck! S From guido at python.org Wed Jan 13 18:58:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:58:04 -0800 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: Memories of days past... Python had several regular expression implementations before, one of which was called "regex". But I would rather not have a new module -- I would much rather have a flag specifying the new (backwards incompatible) syntax/semantics. The flag would have a long name (e.g. re.NEW_SYNTAX), a short name (e.g. re.N) and an inline syntax, "(?n)...". --Guido On Tue, Jan 12, 2010 at 7:58 PM, Brett Cannon wrote: > > > On Tue, Jan 12, 2010 at 14:10, MRAB wrote: >> >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. >> > > If it is a separate module under a different name it can do the proper > thing. People will just need to be aware of the difference when they import > the module. > >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? >> > > That's totally up to you. There is practically no chance of it getting into > the 2.x under the stdlib at this point since 2.7b1 is coming up and this > module has not been out in the wild for a year (to my knowledge). ?If you > want to support 2.x that's fine and I am sure users would appreciate it, but > it isn't necessary to get into the Python 3 stdlib. > >> >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? >> > > If you want to minimize the differences then it should probably match. As I > said, since it is a different name to import under it can deviate where > reasonable, just make sure to clearly document the deviations. > -Brett > >> >> Thanks >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From ralf at brainbot.com Wed Jan 13 19:03:08 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 19:03:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Steven Bethard's message of "Wed, 13 Jan 2010 09:57:29 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <87fx69dgrn.fsf@muni.brainbot.com> Steven Bethard writes: > > How complicated is your PYTHONSTARTUP file? My suspicion is that you > could easily write it to work for both Python 2.X and 3.X. sure. that's exactly what I did. My point is that sharing those environment variables will cause pain for some people. From guido at python.org Wed Jan 13 19:07:58 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:07:58 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. -1. They have their use, but more in controlled situations. If you have "global" env vars that you only want to use with Python 2.x, write a wrapper for Python 3 that invokes it with -E. -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Wed Jan 13 19:14:21 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 13 Jan 2010 13:14:21 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E0D7D.3040806@scottdial.com> On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Wed Jan 13 19:22:33 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:22:33 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Wed, Jan 13, 2010 at 4:24 AM, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> If there is something missing from the stdlib discussion that you think should > be brought up at the summit let me know. And if there is something here you want > to discuss before the summit let me know and I can start a separate thread on it. > > I'm sorry I won't be able to come to the language summit, but I would like if > possible to expedite a pronouncement on PEP 391 (configuring logging using > dictionaries). I believe I addressed all the comments made on the discussion > threads mentioned in the PEP and so I'm not sure what more I need to do to get a > pronouncement. I guess the stdlib slot gives an opportunity for people to air > their views and so I'd be grateful if you added it to the agenda. Is the PEP in the stage of consensus yet? If so it may not even be necessary to discuss it at the summit (though I certainly won't stop people if they want to). -- --Guido van Rossum (python.org/~guido) From phd at phd.pp.ru Wed Jan 13 19:18:50 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 21:18:50 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <20100113181850.GA3837@phd.pp.ru> On Wed, Jan 13, 2010 at 12:57:42PM -0500, ssteinerX at gmail.com wrote: > Or, how about just removing the antiquated use of environment variables "antiquated"? You are kidding, aren't you?! Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Jan 13 19:51:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:51:14 -0800 Subject: [Python-Dev] PyCon Keynote Message-ID: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Jan 13 20:13:58 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:13:58 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D9310.6060907@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> Message-ID: <4B4E1B76.4010309@v.loewis.de> >>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>> X86-64 binary -- does not include source) >>> >>> instead of: >>> >>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>> not include source) >>> >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. >> > > I think we should use whatever is most informative and least confusing > to our users - we owe our allegiance to them and not to a processor vendor. And why do you think this is x86-64? Regards, Martin From martin at v.loewis.de Wed Jan 13 20:33:24 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:33:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA0DA.7070007@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> <4B4DA0DA.7070007@gmail.com> Message-ID: <4B4E2004.8050905@v.loewis.de> > Note that increased 3.x compatibility in the most recent 2.x release > will always help in two scenarios: > > 1. New projects that want to use 2.x only libraries but want to be ready > for the Py3k transition in their own code (e.g. new 2.7 features like > set literals, dict and set comprehensions and the view* dict methods can > be translated far more cleanly by 2to3 than the closest comparable 2.6 > code). Of these, I can only see this being the case of the view* methods. Why would set literals allow for code that more cleanly translates to 3.x? Suppose you write f = set((1,2,3)) in 2.6. That code will work *exactly* the same way in 3.1, no translation needed at all. Likewise for dict and set comprehensions: the code is as cleanly translated in the 2.7 form as it is in any prior form (ie. no modifications). As for view* methods: there is one important exception where the 2.6 equivalent is as cleanly translated as a view method: for key in D: action This is a more modern equivalent for iterating over D.keys(), so if your code still does the latter, just change it to the former (requires Python 2.2). I'd claim that this is the dominant case of traversing a dictionary (probably followed by iterating over .items()). So while it is true that only view* can be transformed correctly, I'd claim that this is an infrequent issue. > 2. Similarly new projects that use a 3.x trunk can be more readily > translated to a 2.7 version with 3to2, whereas some constructs may be > difficult to recreate in earlier Python versions. That may be true, but is besides the point here: the issue was whether a 2.8 release might help in migrating to 3.x. People who follow this approach already have 3.x code. Whether they would rather port to 2.8 only, and get that work with little effort, or whether they would port to 2.5 and later, and put in more work, I don't know - I guess we would have to ask these people. I would expect that they prefer if 2.x died rather sooner than later, because they can then stop supporting it. Regards, Martin From tjreedy at udel.edu Wed Jan 13 20:36:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Jan 2010 14:36:03 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: On 1/13/2010 2:13 PM, "Martin v. L?wis" wrote: >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. > > And why do you think this is x86-64? I do not believe I have personally seen, or at least noticed, that specific term before. Something like "Release for 64-bit Windows (Win NT, 2000, XP, Vista, 7 ) on AMD64-type processors from AMD and Intel" should be clear enough. From martin at v.loewis.de Wed Jan 13 20:40:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 20:40:30 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E21AE.40602@v.loewis.de> > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). Windows actually does - it runs on IA-64 (which is a non-x86 architecture). However, end users are unlikely to use such hardware, so distinguishing between 32-bit and 64-bit is typically good enough. > In this case, making it clear that the 64-bit Windows binaries work for > both Intel and AMD chips is more important than using only the official > architecture name. Well, we did have Itanium binaries at one point, and the "64-bit Windows binaries" you are referring to most definitely don't run on Itanium, even though that's an Intel chip. Regards, Martin From martin at v.loewis.de Wed Jan 13 20:45:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:45:40 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E0D7D.3040806@scottdial.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> Message-ID: <4B4E22E4.4040504@v.loewis.de> > So to echo what Michael said, the Microsoft nomenclature is "x64" > regardless of yours and Martin's objections to that name. Nobody who > uses Windows would be confused by "x64" since that is *the* Microsoft > naming scheme. That's actually not entirely true. There are several places in the APIs where Microsoft either allows or requires to call the architecture AMD64 (e.g. architecture indication in MSI files). Regards, Martin From regebro at gmail.com Wed Jan 13 20:50:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 20:50:59 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. What do you need to do in the PYTHONSTARTUP file? Ten years of Python programming, and I didn't even know it existed. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From phd at phd.pp.ru Wed Jan 13 21:08:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 23:08:40 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <20100113200840.GC14858@phd.pp.ru> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From murman at gmail.com Wed Jan 13 21:12:11 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 14:12:11 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E22E4.4040504@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> <4B4E22E4.4040504@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 13:45, "Martin v. L?wis" wrote: >> So to echo what Michael said, the Microsoft nomenclature is "x64" >> regardless of yours and Martin's objections to that name. Nobody who >> uses Windows would be confused by "x64" since that is *the* Microsoft >> naming scheme. > > That's actually not entirely true. There are several places in the > APIs where Microsoft either allows or requires to call the architecture > AMD64 (e.g. architecture indication in MSI files). I should have clarified I was talking about the names shown on MSDN subscriptions for downloading installation media of Windows 7 and Windows Vista. It's pretty clear in that context Microsoft uses x64 to describe this platform of Windows. But again, it's far from clear that this is a term they use for non-developers. -- Michael Urman From regebro at gmail.com Wed Jan 13 21:27:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 21:27:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113200840.GC14858@phd.pp.ru> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> Message-ID: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: > On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >> What do you need to do in the PYTHONSTARTUP file? >> Ten years of Python programming, and I didn't even know it existed. :-) > > ? See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Cheese and fries! :-) Anyway, I don't see how this could pose any problems, because any user advanced enough to do these things will be advanced enough to understand what the problem is and fix it so it works under Python 3 too. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ralf at brainbot.com Wed Jan 13 21:52:34 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 20:52:34 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> (Lennart Regebro's message of "Wed, 13 Jan 2010 20:50:59 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <874ompg225.fsf@brainbot.com> Lennart Regebro writes: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) hehe. tab completion: # -*- mode: python -*- # Last changed: 2009-12-23 22:25:15 by ralf import sys import os def initreadline(): try: import readline except ImportError: sys.stdout.write("Module readline not available.\n") return import rlcompleter readline.parse_and_bind("tab: complete") # Use tab for completions readline.parse_and_bind('tab: complete') # This forces readline to automatically print the above list when tab # completion is set to 'complete'. readline.parse_and_bind('set show-all-if-ambiguous on') # Bindings for incremental searches in the history. These searches # use the string typed so far on the command line and search # anything in the previous input history containing them. readline.parse_and_bind('"\C-r": reverse-search-history') readline.parse_and_bind('"\C-s": forward-search-history') history = os.path.expanduser("~/.pyhistory") if os.path.exists(history): readline.read_history_file(history) import atexit atexit.register(lambda: readline.write_history_file(history)) initreadline() del initreadline From martin at v.loewis.de Wed Jan 13 22:05:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 22:05:03 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <4B4E357F.4050107@v.loewis.de> Lennart Regebro wrote: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? I personally set up readline: set tab completion, and load the history of commands from the previous session. Of course, I don't know what Ralf uses it for. Regards, Martin From ncoghlan at gmail.com Wed Jan 13 22:43:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 07:43:41 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <4B4E3E8D.2010407@gmail.com> Guido van Rossum wrote: > On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about > just removing the antiquated use of environment variables altogether > from Python 3 and avoid the issue completely. > > -1. They have their use, but more in controlled situations. If you > have "global" env vars that you only want to use with Python 2.x, > write a wrapper for Python 3 that invokes it with -E. Perhaps a case can be made for Python 3 to assume -E by default, with a -e option to enable reading of the environment variables? That way naive users could run Python3 without tripping over existing Python2 environment variables, while other tools could readily set up a different environment before launching Python3. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Wed Jan 13 23:45:57 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 14:45:57 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 1:43 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users using environment variables? That's a recipe for disaster in any version! :-) -- --Guido van Rossum (python.org/~guido) From jared.grubb at gmail.com Wed Jan 13 23:56:37 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Wed, 13 Jan 2010 14:56:37 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <13F6C43A-D62A-4A9F-AF7A-9BDAC94F7D72@gmail.com> On 13 Jan 2010, at 13:43, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. I raised a question about these PYTHON3* variables once before in a discussion about shebang lines: http://www.mailinglistarchive.com/python-dev at python.org/msg29500.html I'm not advocating them, but just wanted to make sure to bring up the shebang use case. Jared From skip at pobox.com Wed Jan 13 23:24:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 16:24:13 -0600 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <19278.18445.428716.321365@montanaro.dyndns.org> Lennart> What do you need to do in the PYTHONSTARTUP file? Just reading off stuff from my own personal startup file... I use it for stuff I want available during interactive sessions: 1. Enable true division. 2. Conditionally define "help" from back in the days when there was no help builtin function. 3. Auto-save session (readline) history so I can easily recall commands across sessions. 4. Add other fake builtins ("like") or override behavior of some (like "dir") making them handier for interactive use. 5. autoload modules/symbols (pokes around in common modules from sys.excepthook function). Oh, and I've had no particular trouble keeping it working in Python 1, 2 or 3. Skip From fuzzyman at voidspace.org.uk Thu Jan 14 01:20:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 00:20:21 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: <4B4E6345.5070202@voidspace.org.uk> On 13/01/2010 19:13, "Martin v. L?wis" wrote: >>>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>>> X86-64 binary -- does not include source) >>>> >>>> instead of: >>>> >>>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>>> not include source) >>>> >>>> >>> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >>> to use AMD64 instead. I think we should comply - they invented the >>> architecture, so they have the right to give it a name. Neither >>> Microsoft nor Intel have such a right. >>> >>> >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. >> > And why do you think this is x86-64? > Well anecdotal everyone I have *every* talked to about 64bit processors has referred to having a 64bit processor (x86 is a given) and not an AMD64 architecture processor. Linus Torvalds addressed this specific issue for Linux and came down on the side of "x86-64": http://kerneltrap.org/node/2466 Look up AMD64 on Wikipedia and it redirects you to the X86-64 page. Information website setup by AMD and partners about the AMD64 architecture: http://www.x86-64.org/about.html In the AMD website they refer to "x86-64 Assembly": http://www.x86-64.org/documentation/assembly.html Microsoft seem to draw a distinction between x64 (which would also be acceptable) and Itanium based systems. Very rarely do they refer to AMD64: * http://www.microsoft.com/servers/64bit/compare.mspx * http://www.microsoft.com/servers/64bit/x64/overview.mspx * http://www.microsoft.com/servers/64bit/overview.mspx Using a vendor specific name automatically begs the question as to whether the installer works on processors from other vendors, as we saw in the specific enquiry from the user that triggered this debate. Referring to the AMD 64 build as x86-64, with a footnote explaining which architectures this specifically means is unlikely to confuse people. It is *definitely* better than just saying AMD64. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Thu Jan 14 02:50:55 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 19:50:55 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? Message-ID: <19278.30847.649228.115514@montanaro.dyndns.org> Just out of curiosity, is the static build stuff (use the old Modules/Setup file to build modules) exercised at all any more? Skip From benjamin at python.org Thu Jan 14 03:22:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 13 Jan 2010 20:22:03 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? In-Reply-To: <19278.30847.649228.115514@montanaro.dyndns.org> References: <19278.30847.649228.115514@montanaro.dyndns.org> Message-ID: <1afaf6161001131822r151bd202x35653ba44ee0235d@mail.gmail.com> 2010/1/13 : > > Just out of curiosity, is the static build stuff (use the old Modules/Setup > file to build modules) exercised at all any more? Exercised as in used in the build process? Yes. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Jan 14 10:23:47 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 09:23:47 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! Message-ID: <954450.26617.qm@web25804.mail.ukl.yahoo.com> In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: http://www.python.org/dev/peps/pep-0391/ In November 2009 I posted to this list that the PEP was ready for review. I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. Thanks & regards, Vinay Sajip From ziade.tarek at gmail.com Thu Jan 14 10:30:15 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Jan 2010 10:30:15 +0100 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <94bdd2611001140130u6154e893jfd37db32a25be66a@mail.gmail.com> On Thu, Jan 14, 2010 at 10:23 AM, Vinay Sajip wrote: [..] > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would > be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check > in the changes unless there are objections to doing so. All those who think that logging is currently > hard to configure will benefit from these changes, so here's the opportunity to pipe up and > improve things. FWIW, I am +1. Thanks for your work Tarek From hodgestar+pythondev at gmail.com Thu Jan 14 10:39:41 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 14 Jan 2010 11:39:41 +0200 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: On Thu, Jan 14, 2010 at 11:23 AM, Vinay Sajip wrote: > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I think the dict configuration scheme will be a great addition to the logging module. :) Schiavo Simon From p.f.moore at gmail.com Thu Jan 14 12:22:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jan 2010 11:22:09 +0000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> 2010/1/14 Vinay Sajip : > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I've no immediate need for the feature, but it would be good to have something like this, so I'm +1. Paul. From ncoghlan at gmail.com Thu Jan 14 12:46:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:46:19 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> Message-ID: <4B4F040B.8010607@gmail.com> Paul Moore wrote: > 2010/1/14 Vinay Sajip : >> So, can you please indicate your vote for or against incorporating PEP 391 into Python? > > I've no immediate need for the feature, but it would be good to have > something like this, so I'm +1. I'm in the same boat as Paul, but PEP 291 provides a nice forward compatible configuration scheme that will work with any application configuration approach that can produce an appropriate Python dictionary. So +1 from me too - I think the PEP has now taken this as far as theorising can go, and the only way to learn anything further is to put it into practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jan 14 12:57:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:57:55 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4B4F06C3.7030806@gmail.com> Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of > configuring logging using dictionaries: > > http://www.python.org/dev/peps/pep-0391/ Although one minor comment: you can probably remove the note about the "ext://" and "cfg://" prefixes being provisional at this stage. Those names look fine to me, so I don't see any point inviting a late-breaking bikeshed discussion about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu Jan 14 14:53:29 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 08:53:29 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> On Thu, Jan 14, 2010 at 4:23 AM, Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: > > ?http://www.python.org/dev/peps/pep-0391/ > > In November 2009 I posted to this list that the PEP was ready for review. > > I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. > > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. > > Thanks & regards, > > Vinay Sajip I'm generally +1 - but given I know that Django 1.2 is slated to implement something somewhat similar, I'm interested to hear how this proposal meshes with their plan(s). jesse From vinay_sajip at yahoo.co.uk Thu Jan 14 15:08:24 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 14:08:24 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> Message-ID: <92210.91656.qm@web25806.mail.ukl.yahoo.com> > From: Jesse Noller > I'm generally +1 - but given I know that Django 1.2 is slated to > implement something somewhat similar, I'm interested to hear how this > proposal meshes with their plan(s).. Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: http://groups.google.com/group/django-developers/msg/4ef81a2160257221 They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. Regards, Vinay Sajip From ssteinerx at gmail.com Thu Jan 14 15:54:27 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 14 Jan 2010 09:54:27 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <62E362ED-5DD7-4D42-B5E0-B263A137FD5C@gmail.com> On Jan 14, 2010, at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. That puts a huge +1 on there for me. If we can get this approved and have Django's logging improved *and* avoid a whole pile of duplicate effort in Django that's a huge win. S From jnoller at gmail.com Thu Jan 14 15:56:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 09:56:18 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <4222a8491001140656w68c7290agccfe7282cf96f2fb@mail.gmail.com> On Thu, Jan 14, 2010 at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. > > Regards, > > Vinay Sajip > Cool, +1 then :) From mal at egenix.com Thu Jan 14 20:19:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 20:19:12 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <4B4F6E30.50400@egenix.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users won't have PYTHONPATH or any other Python environment variables setup. Really, if you know that you are going to run Python 3 instead of Python 2 or vice-versa it's easy enough to run . py3env.sh or . py2env.sh in order to setup your shell environment, much like you typically do when using virtual Python installations or work on different projects that require different setups. If you just want to separate Python 2 and 3 files, use the user site-packages dir which includes the Python version. More experienced users could also write their own environment switching sitecustomize.py or usercustomize.py. And then there's the hackery option for experts that love dirty tricks: add a .pth file which includes an "import mysetup" line to fix-up you path and other settings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jan 14 22:02:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Jan 2010 07:02:28 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <4B4F8664.4080107@gmail.com> M.-A. Lemburg wrote: > Nick Coghlan wrote: >> Guido van Rossum wrote: >>> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >>> just removing the antiquated use of environment variables altogether >>> from Python 3 and avoid the issue completely. >>> >>> -1. They have their use, but more in controlled situations. If you >>> have "global" env vars that you only want to use with Python 2.x, >>> write a wrapper for Python 3 that invokes it with -E. >> Perhaps a case can be made for Python 3 to assume -E by default, with a >> -e option to enable reading of the environment variables? >> >> That way naive users could run Python3 without tripping over existing >> Python2 environment variables, while other tools could readily set up a >> different environment before launching Python3. > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. I was actually thinking of the situation where the OS had a preinstalled Python 2 with a default PYTHONPATH setting and the user was playing with Python 3. However, I agree that that is a fairly unlikely scenario (since preinstalled Pythons tend not to rely on the environment variables and a user sophisticated enough to be playing with a new Python interpreter should be able to cope with a few environment variable conflicts anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Thu Jan 14 22:09:30 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 21:09:30 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F8664.4080107@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <4B4F8664.4080107@gmail.com> Message-ID: <4B4F880A.5010809@voidspace.org.uk> On 14/01/2010 21:02, Nick Coghlan wrote: > However, I agree that that is a fairly unlikely scenario (since > preinstalled Pythons tend not to rely on the e Well, on the other hand I think that during the next few years it will be increasingly common for developers (and possibly users) to have Python 2 and Python 3 installed side-by-side. Many libraries and applications may never make the jump to Python 3 and Python users may be using 'legacy' Python 2 code for many years to come. It will also become increasingly common for developers to be using Python 3 *primarily* and for Python 3 only libraries and applications to emerge. Whilst there are workarounds we *are* in a situation that Python 2 and Python 3 share environment variables for the location of libraries and executing code on startup, whilst at the same time they are largely incompatible and need separate library paths and startup code. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ralf at brainbot.com Thu Jan 14 22:25:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Thu, 14 Jan 2010 22:25:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> (M.'s message of "Thu, 14 Jan 2010 20:19:12 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <871vhswf90.fsf@brainbot.com> "M.-A. Lemburg" writes: > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. > > Really, if you know that you are going to run Python 3 instead of > Python 2 or vice-versa it's easy enough to run You don't even know that you're going to run python. I have 40 python scripts in my /usr/bin directory. > > . py3env.sh > or > . py2env.sh > > in order to setup your shell environment, much like you typically > do when using virtual Python installations or work on different > projects that require different setups. No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > > If you just want to separate Python 2 and 3 files, use the > user site-packages dir which includes the Python version. Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to install those programs in a user's home directory. There are still people running python <2.6. From mal at egenix.com Thu Jan 14 22:51:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 22:51:07 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <871vhswf90.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <871vhswf90.fsf@brainbot.com> Message-ID: <4B4F91CB.2060106@egenix.com> Ralf Schmitt wrote: > "M.-A. Lemburg" writes: > >> >> Naive users won't have PYTHONPATH or any other Python environment >> variables setup. >> >> Really, if you know that you are going to run Python 3 instead of >> Python 2 or vice-versa it's easy enough to run > > You don't even know that you're going to run python. I have 40 python > scripts in my /usr/bin directory. > >> >> . py3env.sh >> or >> . py2env.sh >> >> in order to setup your shell environment, much like you typically >> do when using virtual Python installations or work on different >> projects that require different setups. > > No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > >> >> If you just want to separate Python 2 and 3 files, use the >> user site-packages dir which includes the Python version. > > Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to > install those programs in a user's home directory. There are still > people running python <2.6. Note that you only need to have a different PYTHONPATH setups for Python 2.x/3.x if you plan to run code that: * is not installed in site-packages (most OS shipped code will be found in the resp. system site-packages/ dir) * is not installed in a user site-packages directory (user installed code will typically go there (*)) * uses modules/packages that come in two different versions (one for Python 2.x and one for 3.x) and use the same name for both versions * needs to work in both Python 2.x and 3.x (*) The method for installing code in user site-packages dir is running: python setup.py install --user instead of the standard python setup.py install which install to the system-wide site-packages. BTW: I guess the bzr and mercurial wikis will need to be updated accordingly - at least for users of Python >=2.6. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Jan 14 22:55:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jan 2010 22:55:04 +0100 Subject: [Python-Dev] [ANN] Python 2.5.5 Release Candidate 1. Message-ID: <4B4F92B8.30806@v.loewis.de> On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.5. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.6 (which is 2.6.4 at this point). This releases fixes issues with the logging and tarfile modules, and with thread-local variables. See the detailed release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed. For more information on Python 2.5.5, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.5.5 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin at v.loewis.de Python Release Manager (on behalf of the entire python-dev team) From status at bugs.python.org Fri Jan 15 18:07:26 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Jan 2010 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100115170726.9963A7865F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/08/10 - 01/15/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2536 open (+32) / 16993 closed (+16) / 19529 total (+48) Open issues with patches: 1024 Average duration of open issues: 710 days. Median duration of open issues: 469 days. Open Issues Breakdown open 2501 (+32) pending 34 ( +0) Issues Created Or Reopened (53) _______________________________ PYTHON3PATH environment variable to supersede PYTHONPATH for mul 01/13/10 CLOSED http://bugs.python.org/issue2375 reopened r.david.murray Mark the compiler package as deprecated 01/13/10 http://bugs.python.org/issue6837 reopened ezio.melotti test_distutils failure 01/15/10 CLOSED http://bugs.python.org/issue6961 reopened flox buildbot test_urllib: unsetting missing 'env' variable 01/08/10 CLOSED http://bugs.python.org/issue7026 reopened flox Two float('nan') are not equal 01/08/10 CLOSED http://bugs.python.org/issue7660 created jmfauth compiling ctypes fails with non-ascii path 01/08/10 http://bugs.python.org/issue7661 created pitrou patch time.utcoffset() 01/09/10 http://bugs.python.org/issue7662 created techtonik UCS4 build incorrectly translates cases for non-BMP code points 01/10/10 CLOSED http://bugs.python.org/issue7663 created exarkun python logger does not handle IOError Exception 01/10/10 CLOSED http://bugs.python.org/issue7664 created yateenjoshi test_urllib2 and test_ntpath fail if path contains "\" 01/10/10 http://bugs.python.org/issue7665 created flox test_lib2to3 fails if path contains space 01/10/10 CLOSED http://bugs.python.org/issue7666 created flox test_doctest fails with non-ascii path 01/10/10 http://bugs.python.org/issue7667 created flox buildbot test_httpservers fails with non-ascii path 01/10/10 http://bugs.python.org/issue7668 created flox buildbot test_unicode_file fails with non-ascii path 01/10/10 CLOSED http://bugs.python.org/issue7669 created flox _sqlite3: Block *all* operations on a closed Connection object 01/10/10 http://bugs.python.org/issue7670 created haypo patch test_popen fails if path contains special char like ";" 01/11/10 http://bugs.python.org/issue7671 reopened flox patch _ssl module causes segfault 01/10/10 http://bugs.python.org/issue7672 created ssoria patch audioop: check that length is a multiple of the size 01/11/10 http://bugs.python.org/issue7673 created haypo patch select.select() corner cases: duplicate fds, out-of-range fds 01/11/10 http://bugs.python.org/issue7674 created cdleary help() doesn't accept unicode literals in built in docstrings 01/11/10 http://bugs.python.org/issue7675 created psd patch IDLE shell shouldn't use TABs 01/11/10 http://bugs.python.org/issue7676 created cben distutils, better error message for setup.py upload -sign withou 01/11/10 http://bugs.python.org/issue7677 created illume subprocess.Popen pipeline example code in the documentation is l 01/11/10 http://bugs.python.org/issue7678 created steven.k.wong Warning building 2.7 on OS X 10.6 libintl.h "Present But Cannot 01/12/10 http://bugs.python.org/issue7679 created ssteinerX pythonw crash while attempting to start() a thread object 01/12/10 http://bugs.python.org/issue7680 created dontbugme Incorrect division in [wave.py] 01/12/10 CLOSED http://bugs.python.org/issue7681 created alfps patch, needs review Optimisation of if with constant expression 01/12/10 http://bugs.python.org/issue7682 created sdefresne patch Wrong link in HTML documentation 01/12/10 CLOSED http://bugs.python.org/issue7683 created francescor decimal.py: infinity coefficients in tuples 01/12/10 http://bugs.python.org/issue7684 created skrah minor typo in re docs 01/12/10 CLOSED http://bugs.python.org/issue7685 created mikejs patch redundant open modes 'rbb', 'wbb', 'abb' no longer work on Windo 01/13/10 http://bugs.python.org/issue7686 created ivank Bluetooth support untested 01/13/10 http://bugs.python.org/issue7687 created pitrou TypeError: __name__ must be set to a string object 01/13/10 http://bugs.python.org/issue7688 created frankmillman Pickling of classes with a metaclass and copy_reg 01/13/10 http://bugs.python.org/issue7689 created craigcitro patch Wrong PEP number in importlib module manual page 01/13/10 CLOSED http://bugs.python.org/issue7690 created francescor test_bsddb3 files are not always removed when test fails 01/13/10 CLOSED http://bugs.python.org/issue7691 created flox buildbot Pointless comparision of unsigned integer with zero 01/13/10 CLOSED http://bugs.python.org/issue7692 created Bugger tarfile.extractall can't have unicode extraction path 01/13/10 http://bugs.python.org/issue7693 created pbienst DeprecationWarning from distuils.commands.build_ext needs stackl 01/13/10 http://bugs.python.org/issue7694 created tseaver patch missing termios constants 01/13/10 http://bugs.python.org/issue7695 created conorh Improve Memoryview/Buffer documentation 01/13/10 http://bugs.python.org/issue7696 created flox os.getcwd() should raise UnicodeDecodeError for arbitrary bytes 01/13/10 CLOSED http://bugs.python.org/issue7697 created flox pystack macro in Misc/gdbinit incorrectly uses PyEval_EvalFrame 01/14/10 CLOSED http://bugs.python.org/issue7698 created exarkun patch strptime, strftime documentation 01/14/10 http://bugs.python.org/issue7699 created mikejs patch "-3" flag does not work anymore 01/14/10 CLOSED http://bugs.python.org/issue7700 created flox patch fix output string length for binascii.b2a_uu() 01/14/10 CLOSED http://bugs.python.org/issue7701 created haypo patch Wrong order of parameters of _get_socket in SMTP class in smtpli 01/14/10 http://bugs.python.org/issue7702 created alito Replace buffer()-->memoryview() in Lib/ctypes/test/ 01/14/10 http://bugs.python.org/issue7703 created flox patch Math calculation problem (1.6-1.0)>0.6, python said TRUE 01/14/10 CLOSED http://bugs.python.org/issue7704 created DhaReaL libpython2.6.so is not linked correctly on FreeBSD when threads 01/15/10 http://bugs.python.org/issue7705 created aballier patch, needs review Missing #include guards 01/15/10 http://bugs.python.org/issue7706 created akrpic77 patch multiprocess.Queue operations during import can lead to deadlock 01/15/10 http://bugs.python.org/issue7707 created kripken Conversion of longs to bytes and vice-versa. 01/11/10 http://bugs.python.org/issue1023290 reopened mark.dickinson patch Issues Now Closed (67) ______________________ segfault in curses when calling redrawwin() before refresh() 825 days http://bugs.python.org/issue1266 mark.dickinson "RuntimeError: dictionary changed size during iteration" in Tkin 751 days http://bugs.python.org/issue1658 flox patch Exception exceptions.AttributeError '_shutdown' in 0.6, python said TRUE 0 days http://bugs.python.org/issue7704 tim_one Support for sending multipart form data 2452 days http://bugs.python.org/issue727898 r.david.murray email.MIME*.as_string removes duplicate spaces 1440 days http://bugs.python.org/issue1422094 r.david.murray easy test_parsedate_acceptable_to_time_functions+DST == :-( 1394 days http://bugs.python.org/issue1454285 r.david.murray email module does not complay with RFC 2046: CRLF issue 1196 days http://bugs.python.org/issue1571841 r.david.murray email.FeedParser.BufferedSubFile improperly handles "\r\n" 969 days http://bugs.python.org/issue1721862 r.david.murray patch, easy Top Issues Most Discussed (10) ______________________________ 20 dtoa.c: oversize b in quorem 11 days open http://bugs.python.org/issue7632 18 _ssl module causes segfault 5 days open http://bugs.python.org/issue7672 12 Speed up cPickle's pickling generally 287 days open http://bugs.python.org/issue5683 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 7 days open http://bugs.python.org/issue7658 8 Fatal error on thread creation in low memory condition 27 days open http://bugs.python.org/issue7544 8 Direct calls to PyObject_Del/PyObject_DEL are broken for --with 558 days open http://bugs.python.org/issue3299 7 "-3" flag does not work anymore 1 days closed http://bugs.python.org/issue7700 7 tarfile.extractall can't have unicode extraction path 2 days open http://bugs.python.org/issue7693 7 test_urllib: unsetting missing 'env' variable 0 days closed http://bugs.python.org/issue7026 7 os.path.abspath with unicode argument should call os.getcwdu 542 days open http://bugs.python.org/issue3426 From g.brandl at gmx.net Sat Jan 16 21:05:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 16 Jan 2010 21:05:46 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> Message-ID: Am 13.01.2010 21:27, schrieb Lennart Regebro: > On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: >> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >>> What do you need to do in the PYTHONSTARTUP file? >>> Ten years of Python programming, and I didn't even know it existed. :-) >> >> See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. > > Cheese and fries! :-) > > Anyway, I don't see how this could pose any problems, because any user > advanced enough to do these things will be advanced enough to > understand what the problem is and fix it so it works under Python 3 > too. I'd propose adding a bit of text to the environment variables documentation, and especially about the needs of PYTHONSTARTUP files. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From asmodai at in-nomine.org Sat Jan 16 21:50:56 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 16 Jan 2010 21:50:56 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <874ompg225.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> Message-ID: <20100116205056.GL99670@nexus.in-nomine.org> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >hehe. tab completion: With bpython and ipython available, why would you even want to stick to the 'plain old' interactive interpreter? (Sorry to derail the discussion, but maybe there's more people that have not heard of either or both.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B For ever, brother, hail and farewell... From solipsis at pitrou.net Sat Jan 16 21:57:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Jan 2010 20:57:13 +0000 (UTC) Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: Jeroen Ruigrok van der Werven in-nomine.org> writes: > > -On [20100113 22:13], Ralf Schmitt (ralf brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick to the > 'plain old' interactive interpreter? Why wouldn't we? There are probably many more people using the standard Python prompt than ipython. From ben+python at benfinney.id.au Sat Jan 16 23:41:03 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Jan 2010 09:41:03 +1100 Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: <87y6jxk70g.fsf@benfinney.id.au> Jeroen Ruigrok van der Werven writes: > -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick > to the 'plain old' interactive interpreter? Because those optional extras are not always available, whereas the standard interactive interpreter is always available with CPython distributions. -- \ ?I fly Air Bizarre. You buy a combination one-way round-trip | `\ ticket. Leave any Monday, and they bring you back the previous | _o__) Friday. That way you still have the weekend.? ?Steven Wright | Ben Finney From ncoghlan at gmail.com Sun Jan 17 01:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:22:10 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87y6jxk70g.fsf@benfinney.id.au> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> <87y6jxk70g.fsf@benfinney.id.au> Message-ID: <4B525832.8090904@gmail.com> Ben Finney wrote: > Jeroen Ruigrok van der Werven writes: > >> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >>> hehe. tab completion: >> With bpython and ipython available, why would you even want to stick >> to the 'plain old' interactive interpreter? > > Because those optional extras are not always available, whereas the > standard interactive interpreter is always available with CPython > distributions. I've never even contemplated the idea of installing 3rd party apps for the 4 current development builds (2.x trunk, 2.x maintenance, 3.x trunk, 3.x maintenance) on my home development machine. And of course work suffers from the problem of not being allowed to install arbitrary apps I downloaded from the internet without getting the licensing vetted for commercial acceptability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 01:34:08 2010 From: nad at acm.org (Ned Deily) Date: Sat, 16 Jan 2010 16:34:08 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? Message-ID: I've recently seen a couple of references to 3.1.2 go by in checkins which made me wonder whether dates have been proposed yet for updates to either 3.1 or 2.6. I don't recall seeing any and I didn't see any references in the PEPs. Some advance warning would be nice. I assume that some critical problem could trigger planning for an update on short notice; is there a time-limit trigger as well? -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Jan 17 01:55:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:55:38 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52600A.7060201@gmail.com> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Usually there is one more regular maintenance release of the existing maintenance branches within a few months of the creation of the next version (releases before then are at the discretion of the release manager, and security releases continue for much longer). So take the planned 2.7 and 3.2 release dates and add a couple of months to each one to get the likely release dates for the 2.6.x and 3.1.x maintenance releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sun Jan 17 10:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 17 Jan 2010 10:21:49 +0100 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52D6AD.6090302@v.loewis.de> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Barry was once proposing time-based releases; this idea didn't really catch on. It's the release manager who decides when the next bug fix release will be made, often in response to developers asking for one. Regards, Martin From solipsis at pitrou.net Sun Jan 17 14:00:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 13:00:04 +0000 (UTC) Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: Message-ID: Ned Deily acm.org> writes: > > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. There are a couple of release blockers right now. Once they are fixed or deferred, I think it would be nice to have a 3.1.2. Why do you need "some advance warning" though? From ncoghlan at gmail.com Sun Jan 17 14:40:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 23:40:42 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B53135A.7060104@gmail.com> Antoine Pitrou wrote: > Ned Deily acm.org> writes: >> I've recently seen a couple of references to 3.1.2 go by in >> checkins which made me wonder whether dates have been proposed yet >> for updates to either 3.1 or 2.6. I don't recall seeing any and I >> didn't see any references in the PEPs. Some advance warning would >> be nice. > > There are a couple of release blockers right now. Once they are fixed > or deferred, I think it would be nice to have a 3.1.2. Why do you > need "some advance warning" though? Advance warning does allow interested users that would consider upgrading to schedule time for testing before the maintenance release comes out. This is particularly useful in helping to make a 1-week RC period effective in picking up issues that might otherwise lead to a brown paper bag release to fix major issues that slipped through our own automated test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 19:01:37 2010 From: nad at acm.org (Ned Deily) Date: Sun, 17 Jan 2010 10:01:37 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: <4B53135A.7060104@gmail.com> Message-ID: In article <4B53135A.7060104 at gmail.com>, Nick Coghlan wrote: > Antoine Pitrou wrote: > > Ned Deily acm.org> writes: > >> I've recently seen a couple of references to 3.1.2 go by in > >> checkins which made me wonder whether dates have been proposed yet > >> for updates to either 3.1 or 2.6. I don't recall seeing any and I > >> didn't see any references in the PEPs. Some advance warning would > >> be nice. > > There are a couple of release blockers right now. Once they are fixed > > or deferred, I think it would be nice to have a 3.1.2. Why do you > > need "some advance warning" though? > > Advance warning does allow interested users that would consider > upgrading to schedule time for testing before the maintenance release > comes out. This is particularly useful in helping to make a 1-week RC > period effective in picking up issues that might otherwise lead to a > brown paper bag release to fix major issues that slipped through our own > automated test coverage. That. and resource contention: there are always potential fixes in the pipeline that could or should be bumped in priority if one knows there is a code cutoff approaching. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Sun Jan 17 20:51:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 20:51:58 +0100 Subject: [Python-Dev] Enhancing the shutil module Message-ID: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Hello, For 2.7/3.2, I am in the process of removing modules in Distutils that can be replaced by calls to existing functions in stdlib. For instance, "dir_util" and "file_util" (old modules from the Python 1.x era) are going away in favor of calls to shutil (and os), so the Distutils package gets lighter. Another module I would like to move away from Distutils is "archive_util". It contains helpers to build archives, whether they are zip or tar files. I propose to move those useful functions into shutil, as this seems the most logical place. I also propose to maintain this "shutil" module for now on (no one is declared as a maintainer in maintainers.rst) since Distutils will become a heavy user of its functions. Any objections/opinions ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 20:53:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 19:53:03 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536A9F.5050203@voidspace.org.uk> On 17/01/2010 19:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > > Regards, > Tarek > > I think it's a great idea. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Sun Jan 17 20:55:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2010 11:55:47 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > If it's archive-agnostic then shutil is probably the best place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Great! > Any objections/opinions ? > No objections from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jan 17 21:04:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 20:04:41 +0000 (UTC) Subject: [Python-Dev] Enhancing the shutil module References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: Tarek Ziad? gmail.com> writes: > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. Are these functions portable? Do they rely on external programs? > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. It's ok for me. Regards Antoine. From ziade.tarek at gmail.com Sun Jan 17 21:09:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:09:18 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: > > > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >> >> Hello, >> >> For 2.7/3.2, I am in the process of removing modules in Distutils that >> can be replaced by calls to existing functions in stdlib. For >> instance, "dir_util" and "file_util" (old modules from the Python 1.x >> era) are going away in favor of calls to shutil (and os), so the >> Distutils package gets lighter. >> >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. > > If it's archive-agnostic then shutil is probably the best place. In more details: It allows the creation of gzip, bzip2, tar and zip files through a single API. There's a registry of supported formats and the API is driven by a format identifier. To do the work it uses stdlib's compression modules. Although it tries the "zip" system command as a fallback if the "zipfile" module is not present. (notice that I've removed the support of "compress" (.Z) some time ago) Regards Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 21:15:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 20:15:00 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536FC4.9090304@voidspace.org.uk> On 17/01/2010 20:04, Antoine Pitrou wrote: > Tarek Ziad? gmail.com> writes: > >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. >> > Are these functions portable? Do they rely on external programs? > > I believe that part of the work that Tarek has been doing has been to make these distutils commands use the Python standard library and not depend on external programs. In which case they seem like *excellent* additions to the shutil module. Of course Tarek can speak for himself... Michael >> I also propose to maintain this "shutil" module for now on (no one is >> declared as a maintainer in maintainers.rst) since Distutils will >> become a heavy user of its functions. >> > It's ok for me. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Sun Jan 17 21:43:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:43:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B536FC4.9090304@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> Message-ID: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> On Sun, Jan 17, 2010 at 9:15 PM, Michael Foord wrote: [..] >> Are these functions portable? Do they rely on external programs? >> >> > > I believe that part of the work that Tarek has been doing has been to make > these distutils commands use the Python standard library and not depend on > external programs. In which case they seem like *excellent* additions to the > shutil module. yes, in the past the "tar" files where created using the "tar" command but this has been changed. For a while now, they are portable and they use stdlib code only. A recent addition is to be able to define user/group permissions in the tar files, thanks to Lars' work in the tarfile module. There's one remaining external call for "zip" done if the zip module is not found, but I am happy to remove it and throw an exception if it's not found, and keep the external "zip" call on Distutils side, so shutil stays 100% stdlib-powered. > Of course Tarek can speak for himself... Thanks for explaining it ! :) Regards Tarek -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Sun Jan 17 22:50:52 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sun, 17 Jan 2010 13:50:52 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> Message-ID: <4B53863C.5060304@activestate.com> On 1/17/2010 12:09 PM, Tarek Ziad? wrote: > On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: >> > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >>> >> Another module I would like to move away from Distutils is >>> >> "archive_util". It contains helpers to build archives, whether they >>> >> are zip or tar files. I propose to move those useful functions into >>> >> shutil, as this seems the most logical place. >> > If it's archive-agnostic then shutil is probably the best place. > In more details: > It allows the creation of gzip, bzip2, tar and zip files through a single API. > There's a registry of supported formats and the API is driven by a > format identifier. Will it also allow decompression of the said archive types? Distribute has some utility code to handle zip/tar archives. So does PyPM. This is because the `tarfile` and `zipfile` modules do not "just work" due to several issues. See http://gist.github.com/279606 Take note of the following in the above code: 1) _ensure_read_write_access 2) *File.is_valid 3) ZippedFile.extract ... issue 6510 4) ZippedFile.extract ... issue 6609 5) TarredFile.extract ... issue 6584 6) The way unpack() detects the unpacked directory. -srid From ziade.tarek at gmail.com Sun Jan 17 23:09:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 23:09:29 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B53863C.5060304@activestate.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> <4B53863C.5060304@activestate.com> Message-ID: <94bdd2611001171409j4ec1e0bfj47e59894fdec0d8a@mail.gmail.com> On Sun, Jan 17, 2010 at 10:50 PM, Sridhar Ratnakumar wrote: [..] > Will it also allow decompression of the said archive types? No but it would make sense having this one as well. Distribute/Setuptools' "unpack_archive" (used by easy_install) was implemented using the same principle as Distutils' "make_archive". I can add it as well while I am at it : it has been successfully used for years by easy_install. > Distribute has > some utility code to handle zip/tar archives. So does PyPM. This is because > the `tarfile` and `zipfile` modules do not "just work" due to several > issues. > > See http://gist.github.com/279606 > > Take note of the following in the above code: > > ?1) _ensure_read_write_access > ?2) *File.is_valid > ?3) ZippedFile.extract ... issue 6510 > ?4) ZippedFile.extract ... issue 6609 > ?5) TarredFile.extract ... issue 6584 Looks like some of these are already fixed (I just looked quickly in the bug tracker though) If its not already done and pending, it would be great if you could refactor your fixes into patches for the remaining issues for each one of those modules Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Sun Jan 17 23:56:37 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 17 Jan 2010 17:56:37 -0500 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: <4B52D6AD.6090302@v.loewis.de> References: <4B52D6AD.6090302@v.loewis.de> Message-ID: On Jan 17, 2010, at 4:21 AM, Martin v. L?wis wrote: > Barry was once proposing time-based releases; this idea didn't really > catch on. I'm still a proponent of this, but it doesn't seem like there's enough support for it. > It's the release manager who decides when the next bug fix release will > be made, often in response to developers asking for one. I'm happy to make a 2.6.5 release whenever it makes sense. -Barry From ncoghlan at gmail.com Mon Jan 18 13:40:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Jan 2010 22:40:01 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> Message-ID: <4B5456A1.2080109@gmail.com> Tarek Ziad? wrote: > There's one remaining external call for "zip" done if the zip module > is not found, but I am happy to remove it and throw an exception if > it's not found, and keep the external "zip" call on Distutils side, so > shutil stays 100% stdlib-powered. +1 for that approach. These changes all sound like nice additions to shutil, and hooray for every module that gets adopted by an active maintainer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From masklinn at masklinn.net Mon Jan 18 14:34:14 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 14:34:14 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5456A1.2080109@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> Message-ID: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: > > Tarek Ziad? wrote: >> There's one remaining external call for "zip" done if the zip module >> is not found, but I am happy to remove it and throw an exception if >> it's not found, and keep the external "zip" call on Distutils side, so >> shutil stays 100% stdlib-powered. > > +1 for that approach. These changes all sound like nice additions to > shutil, and hooray for every module that gets adopted by an active > maintainer :) Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. From doug.hellmann at gmail.com Mon Jan 18 14:46:05 2010 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 18 Jan 2010 08:46:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils >>> side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil > advertises itself as "a number of high-level operations on files and > collections of files." and from what I understood it was a bunch of > shell-type utility functions to easily copy, move or remove files > and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/ > objects in a new module separate from shutil, dealing specifically > with cross-archive APIs and linked from the current archive-specific > modules (essentially, just take the current archive_util, move it to > the toplevel of the stdlib and maybe rename it)? It would also make > the module much easier to find when searching through the module > listing, I think. +1 From fuzzyman at voidspace.org.uk Mon Jan 18 14:57:49 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 18 Jan 2010 13:57:49 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> Message-ID: <4B5468DD.5040200@voidspace.org.uk> On 18/01/2010 13:46, Doug Hellmann wrote: > > On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > >> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>> >>> Tarek Ziad? wrote: >>>> There's one remaining external call for "zip" done if the zip module >>>> is not found, but I am happy to remove it and throw an exception if >>>> it's not found, and keep the external "zip" call on Distutils side, so >>>> shutil stays 100% stdlib-powered. >>> >>> +1 for that approach. These changes all sound like nice additions to >>> shutil, and hooray for every module that gets adopted by an active >>> maintainer :) >> >> Isn't it a bit weird to include that to shutil though? shutil >> advertises itself as "a number of high-level operations on files and >> collections of files." Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> and from what I understood it was a bunch of shell-type utility functions like tar and zip? :-) >> to easily copy, move or remove files and directories (that's pretty >> much all there is in it at this time). >> I don't think the additions are out of place prima-facie. >> Wouldn't it make more sense to put those "archive utils" >> functions/objects in a new module separate from shutil, dealing >> specifically with cross-archive APIs and linked from the current >> archive-specific modules (essentially, just take the current >> archive_util, move it to the toplevel of the stdlib and maybe rename >> it)? It would also make the module much easier to find when searching >> through the module listing, I think. > > +1 > Proliferation of modules is itself a bad thing though. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Mon Jan 18 15:34:12 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 15:34:12 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <94bdd2611001180634sacd998fm6f67dc680e2e61c3@mail.gmail.com> On Mon, Jan 18, 2010 at 2:57 PM, Michael Foord wrote: [..] >>> Wouldn't it make more sense to put those "archive utils" >>> functions/objects in a new module separate from shutil, dealing specifically >>> with cross-archive APIs and linked from the current archive-specific modules >>> (essentially, just take the current archive_util, move it to the toplevel of >>> the stdlib and maybe rename it)? It would also make the module much easier >>> to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. I am with Michael here. I think having this function in shutil is the right place. For the find problem, I think docs.python.org is easy to search now, as long as the shutil module documentation has an good set of examples for this new API. We could even add references in the tarfile, zipfile modules documentation pointing to these examples. Regards Tarek -- Tarek Ziad? | http://ziade.org From masklinn at masklinn.net Mon Jan 18 15:56:05 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 15:56:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> On 18 Jan 2010, at 14:57 , Michael Foord wrote: > > On 18/01/2010 13:46, Doug Hellmann wrote: >> >> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: >> >>> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>>> >>>> Tarek Ziad? wrote: >>>>> There's one remaining external call for "zip" done if the zip module >>>>> is not found, but I am happy to remove it and throw an exception if >>>>> it's not found, and keep the external "zip" call on Distutils side, so >>>>> shutil stays 100% stdlib-powered. >>>> >>>> +1 for that approach. These changes all sound like nice additions to >>>> shutil, and hooray for every module that gets adopted by an active >>>> maintainer :) >>> >>> Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." > > Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? > Well no those are high-level operations on a very restricted set of file types (archives). >>> and from what I understood it was a bunch of shell-type utility functions > > like tar and zip? :-) > Tar and zip have a module each at this time, so they're considered pretty big. I don't think anybody would consider "mv" big enough to give it a module on its own. >>> Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. From phd at phd.pp.ru Mon Jan 18 16:03:38 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 18 Jan 2010 18:03:38 +0300 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118150337.GA19391@phd.pp.ru> On Mon, Jan 18, 2010 at 02:34:14PM +0100, Masklinn wrote: > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. +1 Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ziade.tarek at gmail.com Mon Jan 18 16:24:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 16:24:37 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> Message-ID: <94bdd2611001180724u7f48e620ub32e13ef96c873b9@mail.gmail.com> On Mon, Jan 18, 2010 at 3:56 PM, Masklinn wrote: [..] >> Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> > Well no those are high-level operations on a very restricted set of file types (archives) not really: make_archive/unpack_archive, are also dealing with files and directories. [..] >> Proliferation of modules is itself a bad thing though. > Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. > > Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. I am not sure why this would happen. For instance, shutil is already on the top of the os module since a few major versions IIRC, because it reads and writes files and directories. But it was not moved into the os package (or vice-versa) The shutil module uses APIs to read and write files. So if it works with archives, it's just a specific read/write API that is used, but that doesn't mean tarfile and zipfile might be reunited with shutil imho. If the shutil module is restricted to high-level files and directories manipulation, working with archives is just a target like another I think. But at the end I am 0- to create a new module, because what really matters to me is to take it out of Distutils :) Regards, Tarek -- Tarek Ziad? | http://ziade.org From listsin at integrateddevcorp.com Mon Jan 18 16:56:05 2010 From: listsin at integrateddevcorp.com (Steve Steiner (listsin)) Date: Mon, 18 Jan 2010 10:56:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. As much of a pain as it is to get new modules accepted, I agree that mixing archiving functions into shutil is not the right way to do it and that a separate archive_util module would make much more sense and would give a logical place to put any extensions to archive handling. S From rdmurray at bitdance.com Mon Jan 18 20:28:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 18 Jan 2010 14:28:47 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: > As much of a pain as it is to get new modules accepted, I agree that > mixing archiving functions into shutil is not the right way to do it > and that a separate archive_util module would make much more sense and > would give a logical place to put any extensions to archive handling. Looking at the source code and API for both shutil and archive_util, I think that the archive_util methods fit into shutil. shutil currently wraps some standard library facilities with convenience functions for operations you might otherwise perform at the shell command line using OS facilities. As far as I can tell, archive_util does the same, and seems quite within the shutil mission of "high level file operations". So +1 from me for putting these in shutil. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From p.f.moore at gmail.com Mon Jan 18 20:48:43 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 18 Jan 2010 19:48:43 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> Message-ID: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> 2010/1/18 R. David Murray : > On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: >> As much of a pain as it is to get new modules accepted, I agree that >> mixing archiving functions into shutil is not the right way to do it >> and that a separate archive_util module would make much more sense and >> would give a logical place to put any extensions to archive handling. > > Looking at the source code and API for both shutil and archive_util, I > think that the archive_util methods fit into shutil. ?shutil currently > wraps some standard library facilities with convenience functions > for operations you might otherwise perform at the shell command line using > OS facilities. ?As far as I can tell, archive_util does the same, and > seems quite within the shutil mission of "high level file operations". > > So +1 from me for putting these in shutil. Conceptually, I'm happy with these going into shutil (and +1 on the rest of Tarek's proposal, too!) To my mind, shutil is a module for higher-level operations on files - the sort of things you'd do in shell commands, like move a batch of files around (mv), create a directory tree (mkdir -p). Tarring or zipping up a batch of files fits nicely into that space. Paul. From david.lyon at preisshare.net Tue Jan 19 02:53:43 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 18 Jan 2010 20:53:43 -0500 Subject: [Python-Dev] PyCon Keynote In-Reply-To: References: Message-ID: <0dfb370163cf3a284ca256f49662db74@preisshare.net> On Wed, 13 Jan 2010 10:51:14 -0800, Guido van Rossum wrote: > Please mail me topics you'd like to hear me talk about in my keynote > at PyCon this year. As a typical (but perphaps more vocal) python developer I would like to hear things that might aspire me and others to move to python 3. The way I see it is that python 2.x represents everyones comfort zone. It's safe and nobody got fired at work for using python 2.x I would like to hear how python 3 could be 'shaken' up with a slightly less conservative policy that has gone with the python 2.x series. The logic perhaps being that since only a minority use the 3.x series, it's functionality set is still more up in the air. imo it needs bigger batteries to give it more power than the 2.x series. This meaning that the stdlib could take an extra 5-10 packages not in the 2.x series. Just as one example, sphinx - a great documentation tool. I can easily name five or six others. I'd also like to hear more of your ideas on pypi and getting it to have as much who-ha as CPAN. You could do a lot to enlarge the developer group. Help us all get our priorities to be inline with your own wishes and expectations. If you ask us all to put in a big year and buy you a beer at the end.. I'm certain we all will. Every little bit of inspiration and direction counts for a lot... Best Regards David From ncoghlan at gmail.com Tue Jan 19 12:20:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Jan 2010 21:20:05 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> Message-ID: <4B559565.8090602@gmail.com> Paul Moore wrote: > 2010/1/18 R. David Murray : >> So +1 from me for putting these in shutil. > > Conceptually, I'm happy with these going into shutil (and +1 on the > rest of Tarek's proposal, too!) > > To my mind, shutil is a module for higher-level operations on files - > the sort of things you'd do in shell commands, like move a batch of > files around (mv), create a directory tree (mkdir -p). Tarring or > zipping up a batch of files fits nicely into that space. This is also reflected in the way at least Windows handles archives these days - it took them a couple of iterations to get it right (and resolve some of the performance impacts), but Explorer now does a decent job of integrating archives into the directory tree as "folders that happen to be compressed". Are archives as fundamental as directories and files? No. But in the context of shutil, the fact that their internal structure is largely about directories and files makes them more than just another arbitrary file type. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 19 14:16:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 08:16:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad Message-ID: <20100119081639.5d431ed9@freewill> I've just updated the Launchpad mirrors for the 4 active Python branches, trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar branches on code.python.org but it's probably been 7 months or so since those were regularly updated. Now the Launchpad branches sync against the read-only Subversion branches at http://svn.python.org, so they should remain up-to-date (within the re-sync timeframe of about 4 hours). This means you can once again use Bazaar to get local branches of Python, and you can of course push your own branches to Launchpad. I believe you can even use the bzr-svn plugin to commit changes back to the Subversion master, though I have not yet tried this. To get a local branch, just do any of the following: % bzr branch lp:python (for trunk) % bzr branch lp:python/2.6 % bzr branch lp:python/py3k % bzr branch lp:python/3.1 (It's fairly easy to create new mirrors for other Subversion branches, e.g. Python 2.5; just drop me an email if you want them.) If you're going to create a lot of branches you probably want to put them in a shared repository. E.g. % bzr init-repo pythonbzr % cd pythonbzr % bzr branch lp:python/py3k Bazaar 2.0 or better is recommended. For me, it took about 5m to check the first branch out from Launchpad, and then about 30s or so for each subsequent branch. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From abpillai at gmail.com Tue Jan 19 14:37:18 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Tue, 19 Jan 2010 19:07:18 +0530 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <8548c5f31001190537l2bcab5cfkb6f461910e4abd8b@mail.gmail.com> On Mon, Jan 18, 2010 at 1:21 AM, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > +1 for this. Just make sure that you change the docstring of shutil which now reads as, " shutil - Utility functions for copying files and directory trees." According to this "definition", archives don't fit in there. But the functionality does fit right in, so just need to make sure that it is reflected in the __doc__ . > Regards, > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jan 19 16:50:42 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Jan 2010 15:50:42 +0000 (UTC) Subject: [Python-Dev] Mailing List archive corruption? Message-ID: Hi, When I look at the mailing list archive for python-dev, I see some odd stuff at the bottom of the page: http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 Anyone know what's happened? Regards, Vinay Sajip From barry at python.org Tue Jan 19 17:07:48 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:07:48 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: Message-ID: <20100119110748.69bc564a@freewill> On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: >When I look at the mailing list archive for python-dev, I see some odd stuff at >the bottom of the page: > >http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >Anyone know what's happened? WTF? I think the archives were recently regenerated, so there's probably a fubar there. CC'ing the postmasters. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From foom at fuhm.net Tue Jan 19 17:24:57 2010 From: foom at fuhm.net (James Y Knight) Date: Tue, 19 Jan 2010 11:24:57 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119110748.69bc564a@freewill> References: <20100119110748.69bc564a@freewill> Message-ID: On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > >> When I look at the mailing list archive for python-dev, I see some >> odd stuff at >> the bottom of the page: >> >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 >> >> Anyone know what's happened? > > WTF? I think the archives were recently regenerated, so there's > probably a > fubar there. CC'ing the postmasters. That happens if messages had unescaped "From" lines in the middle of them. No doubt, you've now broken every link anyone had ever made into the python-dev archives, because now all the article numbers are different. BTDT...unfortunately... Pipermail really is quite crappy, sigh. Anyhow, when I did that, I went back to a backup to get the original article numbers, and edited the mbox file escaping From lines or adding additional empty messages until the newly regenerated article numbers matched the originals. I'd highly recommend going through that painful process, since I suspect a *lot* of people have links to the python-dev archive. Hope you have a backup (or can find caches on google or archive.org or something). James From barry at python.org Tue Jan 19 17:44:21 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:44:21 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119114421.3b96fbd4@freewill> On Jan 19, 2010, at 11:24 AM, James Y Knight wrote: >No doubt, you've now broken every link anyone had ever made into the >python-dev archives, because now all the article numbers are >different. BTDT...unfortunately... Pipermail really is quite crappy, >sigh. I've been trying for 10+ years to get folks interested in helping me fix this (and a few other warts) in Pipermail, to not much success. ;/ >Anyhow, when I did that, I went back to a backup to get the original >article numbers, and edited the mbox file escaping From lines or >adding additional empty messages until the newly regenerated article >numbers matched the originals. I'd highly recommend going through that >painful process, since I suspect a *lot* of people have links to the >python-dev archive. Hope you have a backup (or can find caches on >google or archive.org or something). bin/cleanarch uses a set of heuristics to find unescaped From lines and fix them. It's generally pretty good, but it certain can change message numbers (and sadly, their urls). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rdmurray at bitdance.com Tue Jan 19 18:48:41 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 19 Jan 2010 12:48:41 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119174841.9BC3B16A53@kimball.webabinitio.net> On Tue, 19 Jan 2010 11:24:57 -0500, James Y Knight wrote: > > On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > > > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > > > >> When I look at the mailing list archive for python-dev, I see some > >> odd stuff at the bottom of the page: > >> > >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >> > >> Anyone know what's happened? > > > > WTF? I think the archives were recently regenerated, so there's > > probably a fubar there. CC'ing the postmasters. > > That happens if messages had unescaped "From" lines in the middle of > them. > > No doubt, you've now broken every link anyone had ever made into the > python-dev archives, because now all the article numbers are > different. BTDT...unfortunately... Pipermail really is quite crappy, > sigh. > > Anyhow, when I did that, I went back to a backup to get the original > article numbers, and edited the mbox file escaping From lines or > adding additional empty messages until the newly regenerated article > numbers matched the originals. I'd highly recommend going through that > painful process, since I suspect a *lot* of people have links to the > python-dev archive. Hope you have a backup (or can find caches on > google or archive.org or something). The Python issue tracker does, for one. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From ncoghlan at gmail.com Tue Jan 19 22:18:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 07:18:52 +1000 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119174841.9BC3B16A53@kimball.webabinitio.net> References: <20100119110748.69bc564a@freewill> <20100119174841.9BC3B16A53@kimball.webabinitio.net> Message-ID: <4B5621BC.4070608@gmail.com> R. David Murray wrote: > The Python issue tracker does, for one. And all the PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at pythontest.org Wed Jan 20 00:16:44 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 10:16:44 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119081639.5d431ed9@freewill> References: <20100119081639.5d431ed9@freewill> Message-ID: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Hi Barry, That looks very interesting... So does that mean we could update the stdlib for a given python version using this ? David > I've just updated the Launchpad mirrors for the 4 active Python branches, > trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar > branches > on code.python.org but it's probably been 7 months or so since those were > regularly updated. Now the Launchpad branches sync against the read-only > Subversion branches at http://svn.python.org, so they should remain > up-to-date > (within the re-sync timeframe of about 4 hours). > > This means you can once again use Bazaar to get local branches of Python, > and > you can of course push your own branches to Launchpad. I believe you can > even > use the bzr-svn plugin to commit changes back to the Subversion master, > though > I have not yet tried this. > > To get a local branch, just do any of the following: > > % bzr branch lp:python (for trunk) > % bzr branch lp:python/2.6 > % bzr branch lp:python/py3k > % bzr branch lp:python/3.1 > > (It's fairly easy to create new mirrors for other Subversion branches, > e.g. Python 2.5; just drop me an email if you want them.) > > If you're going to create a lot of branches you probably want to put them > in a > shared repository. E.g. > > % bzr init-repo pythonbzr > % cd pythonbzr > % bzr branch lp:python/py3k > > Bazaar 2.0 or better is recommended. For me, it took about 5m to check > the > first branch out from Launchpad, and then about 30s or so for each > subsequent > branch. > > Enjoy, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/david.lyon%40pythontest.org > From barry at python.org Wed Jan 20 00:54:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 18:54:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119185439.299660c7@freewill> On Jan 20, 2010, at 10:16 AM, David Lyon wrote: >Hi Barry, > >That looks very interesting... Hi David, >So does that mean we could update the stdlib for a given >python version using this ? In a sense, yes (if I understand your question correctly). You can use Bazaar to branch any of the 4 Python series and use all the modern DVCS goodness to develop your updates. You can share your branches with others via e.g. Launchpad and even request reviews (called "merge proposals") to get feedback from others. The one thing I am unsure about, mostly because I have not tried it, is whether your Bazaar branch can be used to commit directly back to the Python Subversion master branches. I /think/ the answer is yes, assuming of course that you have permission to do so, and that you have a modern version of Bazaar and the bzr-svn plugin. It might even be possible to commit your Bazaar branch to a local Subversion branch, and then commit the latter to get it pushed up to svn.python.org. At worst, you would use Bazaar's features to get your patch into a state you and your reviewers are happy with, then you would generate a diff for application to your copy of the svn.python.org branch. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 01:51:23 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 11:51:23 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119185439.299660c7@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> Message-ID: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> > On Jan 20, 2010, at 10:16 AM, Barry wrote: >> So does that mean we could update the stdlib for a given >> python version using this ? > > In a sense, yes (if I understand your question correctly). Yeah, it just needs an implementation. > The one thing I am unsure about, mostly because I have not tried it, is > whether your Bazaar branch can be used to commit directly back to the > Python Subversion master branches. I /think/ the answer is yes, > assuming of course that you have permission to do so... Well I'm too Senior and my stuff is too forward looking to qualify for that just yet. I'd be happy to see bzr and mercurial and git all made it together into the stdlib for python 3. That would give a superb updating mechanism for python that would propel python well beyond the dinosaur badlands of CPAN and other languages. I was actually reading from (http://en.wikipedia.org/wiki/Python_%28programming_language%29): "Rather than requiring all desired functionality to be built into the language's core, Python was designed to be highly extensible. .. .. This design of a small core language with a large standard library and an easily extensible interpreter was intended by Van Rossum from the very start because of his frustrations with ABC (which espoused the opposite mindset).[5]" To me, the source code control systems seem to be fully in tune with the original design of python. That is, to be able to easily pull external libraries in. I think what has changed is that the mechanisms now (the SCMs) are way more highly developed than before. Apart from that though, after reading the full wikipedia article I'm left with the distinct impression that things are still pretty much the same (in that python design philosophy is advanced), just that the landscape (of external C libraries) has changed. Now all the libraries are external (on the internet) and all externally managed. So with just a tiny amount of work, imho we could pull it all together to bring python 3 *back* to being that cool tool that it once was (not saying it isn't now). Were you offering me an experimental branch somewhere for python 3 SCM integration ? David From jnoller at gmail.com Wed Jan 20 02:09:15 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 20:09:15 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> On Tue, Jan 19, 2010 at 7:51 PM, David Lyon wrote: >> On Jan 20, 2010, at 10:16 AM, Barry wrote: > >>> So does that mean we could update the stdlib for a given >>> python version using this ? >> >> In a sense, yes (if I understand your question correctly). > > Yeah, it just needs an implementation. > >> The one thing I am unsure about, mostly because I have not tried it, is >> whether your Bazaar branch can be used to commit directly back to the >> Python Subversion master branches. ?I /think/ the answer is yes, >> assuming of course that you have permission to do so... > > Well I'm too Senior and my stuff is too forward looking to qualify > for that just yet. > > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. i sincerely doubt that a source control system will be included in the standard library in the future. Especially 3. A SCM is not a "package management system". Barry was talking about mirrors of the python code. It is true a "package manager" could be developed based on a SCM, however you need to implement this far away from the stdlib and get traction with it within the community long before inclusion would be considered. The decision to move python's source control from SVN to mercurial was controversial enough; including 3 or more scm libraries into core would be an intractable uphill mountain of bike sheds. > So with just a tiny amount of work, imho we could pull > it all together to bring python 3 *back* to being that > cool tool that it once was (not saying it isn't now). Python 3 is still modularized, still has a standard library, etc. If you're really interested in helping with the standard library, get on stdlib-sig, and get ready to write code and PEPs. > Were you offering me an experimental branch somewhere > for python 3 SCM integration ? Barry made bzr mirrors of the python svn tree. Not a python with bzr included. jesse From barry at python.org Wed Jan 20 04:32:30 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 22:32:30 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <20100119223230.4c4a7ed5@freewill> On Jan 19, 2010, at 08:09 PM, Jesse Noller wrote: >The decision to move python's source control from SVN to mercurial was >controversial enough; including 3 or more scm libraries into core >would be an intractable uphill mountain of bike sheds. I'd be surprised if any of the big 3 DVCS developers would actually /want/ their stuff in the stdlib. Being in the stdlib has its advantages and disadvantages. I think for rapidly developing technology, the latter can actually outweigh the former. (Besides, git in the stdlib doesn't make much sense :). >Barry made bzr mirrors of the python svn tree. Not a python with bzr included. Bingo. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 04:43:12 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 14:43:12 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> > On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: > Python 3 is still modularized, still has a standard library, etc. If > you're really interested in helping with the standard library, get on > stdlib-sig, and get ready to write code and PEPs. Thank you for your direction to move these items forward to PEPs and Code. > i sincerely doubt that a source control system will be included in the > standard library in the future. Especially 3. Yeah and who twenty years ago thought you would get a 1GB memory card for $3 when all we had was 10Meg hard disks and they were the full 8" platter. > A SCM is not a "package management system". Exactly. It almost makes the need for a "package management system" pretty much obsolete if you can update your code directly from the developers sources. That's what all these SCMs provide. Plus it's addictive. It's hard to go back to 'package' style technology once you have all your code on an SCM based feed. > Barry was talking about mirrors of the python code. It is true a > "package manager" could be developed based on a SCM, however you need > to implement this far away from the stdlib and get traction with it > within the community long before inclusion would be considered. I think I'll have better chances with PEPs. Being honest, if wonderful libraries like Sphinx and Mercurial and Git and BZR can't make it into the stdlib, then there is no hope for even newer code to get in there. Plus, promoting all sorts of new and fangled tools however good they may or may not be just confuses users and ends up being a waste of time imho. It isn't good management of volounteers time and effort. If you could imagine disaster relief coordinated this way, it would just be a disaster in itself. That's why it has taken some 5 years to get PEP-345 done. > The decision to move python's source control from SVN to mercurial was > controversial enough; including 3 or more scm libraries into core > would be an intractable uphill mountain of bike sheds. Not at all. It would be a very fair thing to do. Not to mention being great for users. > Barry made bzr mirrors of the python svn tree. Not a python with bzr > included. I can't resist asking for that again.. I heard it only in Monty speek. Did you just say ?: "Barry made a bizarre mirror of the python suvern tree. Not a python with a buzzer included." Anyway.. Maybe I do get what your talking about. Even if you do talk with a strange accent. :-) David From jnoller at gmail.com Wed Jan 20 05:10:07 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 23:10:07 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001192010v66b728dbxa7ad1ed1fc1df15f@mail.gmail.com> On Tue, Jan 19, 2010 at 10:43 PM, David Lyon wrote: [snip] > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. Did you ever stop to think that some package authors do not want their code in the standard library? That throwing random shiny things in there just makes it a junk drawer? Besides: Show us the PEP to include Sphinx, and it's dependencies in the standard lib, with Georg's signature at the bottom. The authors of modules have to want things to be in there, they have to be best of breed, tested or common-enough patterns to warrant a slot. We have things in the standard lib which still need more TLC and maintenance, or they need to be shunted into space. Adding random things, which may or may not help packaging, and installing things from random places in someone source repository won't make things better. > Plus, promoting all sorts of new and fangled tools however > good they may or may not be just confuses users and ends > up being a waste of time imho. It isn't good management > of volounteers time and effort. > > If you could imagine disaster relief coordinated this > way, it would just be a disaster in itself. > > That's why it has taken some 5 years to get PEP-345 done. I'm going to assume that you're trolling now, or intentionally misrepresenting facts. Maybe a little of both. A PEP, and an implementation and the ability to rationally debate, discuss and defend your proposal is what is needed to enact changes on policies, python-core or the standard library. >> The decision to move python's source control from SVN to mercurial was >> controversial enough; including 3 or more scm libraries into core >> would be an intractable uphill mountain of bike sheds. > > Not at all. > > It would be a very fair thing to do. Not to mention being > great for users. There should be one-- and preferably only one --obvious way to do it. > Anyway.. Maybe I do get what your talking about. Even if you do > talk with a strange accent. :-) My sense of humor has been disabled by repeated stunning at your hands. I admire your enthusiasm, even if I do think some of it is misplaced, or at guided into the proper channels at very least. Please, you seem to have the time and willingness to help, please go about this the right way. Discuss things on the proper lists, make concrete proposals. If you have have standard lib changes, discuss them on stdlib-sig, if you have ideas about python-the-language, or the interpreter, etc - please discuss it on python-ideas. jesse From ben+python at benfinney.id.au Wed Jan 20 05:16:25 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 15:16:25 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <87wrzdfm1y.fsf@benfinney.id.au> "David Lyon" writes: > Being honest, if wonderful libraries like Sphinx and Mercurial and Git > and BZR can't make it into the stdlib, then there is no hope for even > newer code to get in there. Those are applications, not libraries. Applications don't belong in the standard library. -- \ ?If you pick up a starving dog and make him prosperous, he will | `\ not bite you. This is the principal difference between a dog | _o__) and a man.? ?Mark Twain, _Pudd'n'head Wilson_ | Ben Finney From brian.curtin at gmail.com Wed Jan 20 05:19:47 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 19 Jan 2010 22:19:47 -0600 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: On Tue, Jan 19, 2010 at 21:43, David Lyon wrote: > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > I'm not entirely sure I see why the inclusion of a SCM into the stdlib is necessary. Just because pieces of software are mature and proven in their fields doesn't mean we should add them, or that them *not* being in the stdlib should be a basis for other projects making it in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Jan 20 05:26:44 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:26:44 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119232644.7fa25691@freewill> On Jan 20, 2010, at 02:43 PM, David Lyon wrote: >> On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: >> A SCM is not a "package management system". > >Exactly. It almost makes the need for a "package management system" >pretty much obsolete if you can update your code directly from >the developers sources. > >That's what all these SCMs provide. Plus it's addictive. It's >hard to go back to 'package' style technology once you have >all your code on an SCM based feed. Well... I'm not so sure. A package management system like apt does a /ton/ of additional bookkeeping and work to ensure a robust, highly consistent, functioning system. And while both Python and most Linux distributions have their own notion of "package management", they don't always play nicely together. Tarek and the distutils-sig's work is trying to make the world a better place by bridging this gap better, and there is code out there that makes it easier to say import a Python package from the Cheeseshop and .deb-ify it for use on Debian and Ubuntu. There's also work being done in Launchpad that will allow you to "build-from-branch" so that in a sense you could let a build farm take your Bazaar branches and automatically build the packages from them. I've strayed off-topic I suppose, but I see SCMs and package managers as complementary technologies that help with important parts of the process of delivering software to end-users, but I don't quite see how one can make the other obsolete. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 05:29:34 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 15:29:34 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> > On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: > > I'd be surprised if any of the big 3 DVCS developers would actually /want/ > their stuff in the stdlib. If they ask, they'll get told they're motorbike-shedding. "It's better if their users ask". So here I am as a user doing things the 'right' way. > Being in the stdlib has its advantages and > disadvantages. I think for rapidly developing technology, the > latter can actually outweigh the former. If it's about being able to do updates, then I think this resolves an old and circular argument. As the SCM implementation would, one would expect, to be able to update itself. Side benefits are that it can update everything else along with it at the same time. User Apps, Packages, whatever. It's even better having SCM in an Industrial/Scientific environment. Here's an example: - a machine breaks.. (I mean the software for/in it) - you fix the code, maybe on the spot - you commit and push back to the repository - your code gets checked in and run through the testbot and then you get blamed and have to do the whole thing again properly with a test case. Oh well.. Well anyway, whatever you guys might say, that's a whole lot more efficient than running back to the development machine and going through some obscure build and test and publish process to do a fix on a production machine. Point : The fact that SCMs are two way is great in a production environment. No packaging solution can come close. So why not have python SCMs included as batteries in python.. All these arguments I can take off to the stdlib list when I get the chance.. David From barry at python.org Wed Jan 20 05:50:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:50:36 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119235036.57f6e867@freewill> Okay, last follow up on this and then I'm going to bed. :) On Jan 20, 2010, at 03:29 PM, David Lyon wrote: >> On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: >> >> I'd be surprised if any of the big 3 DVCS developers would actually /want/ >> their stuff in the stdlib. > >If they ask, they'll get told they're motorbike-shedding. "It's better >if their users ask". So here I am as a user doing things the 'right' >way. Actually, you're not. It's not up to the Python community to initiate this. If you really want this, you should engage with the relevant DVCS communities and push them to request it. >Side benefits are that it can update everything else along >with it at the same time. User Apps, Packages, whatever. I get that. Heck, I still run one Gentoo server which I think is as close to the edge you're describing as I'm comfortable with. It's all great until the wheels come off and then it can take *days* to get a functioning system again. The big difference is that I rely on my DVCS to keep one small thing, or a few variants of the same thing, all sane. But I rely on my distribution vendor to keep a thousand complex, interdependent, interacting, sometimes conflicting things sane and working. >Point : The fact that SCMs are two way is great in > a production environment. No packaging solution > can come close. Try talking with some hard-core operations guys, the folks with the keys to the data centers, who work tireless, insanely hours keeping incredibly complex systems running with very little downtime. I think you'd get a different perspective to put it mildly. :) to-sleep-perchance-to-dream-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 06:10:15 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 16:10:15 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <87wrzdfm1y.fsf@benfinney.id.au> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> Message-ID: <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> > "David Lyon" writes: > >> Being honest, if wonderful libraries like Sphinx and Mercurial and Git >> and BZR can't make it into the stdlib, then there is no hope for even >> newer code to get in there. > > Those are applications, not libraries. Applications don't belong in the > standard library. Haha funny.. Well using that logic, distutils is an application.. Are you saying that distutils should be removed? That is most certainly an application. Lets not get too pedantic here. Mercurial and bzr have a built in API that can be called in a library like way. It's true they also have a command line interface in the same way that distutils does. I'm not saying anything negative about distutils. Given that Tarek has an upcoming Pycon presentation where the program talks about a distutils revamp. I'm hoping that he can find some young 20 yr olds and put a cool web interface on that thing. Given that there are empty sprints at pycon. It couldn't hurt to throw that challenge out. Anyway, we'll see.. David From ben+python at benfinney.id.au Wed Jan 20 06:32:10 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:32:10 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> <20100119235036.57f6e867@freewill> Message-ID: <87iqaxfijp.fsf@benfinney.id.au> Barry Warsaw writes: > On Jan 20, 2010, at 03:29 PM, David Lyon wrote: > >So here I am as a user doing things the 'right' way. > > Actually, you're not. It's not up to the Python community to initiate > this. If you really want this, you should engage with the relevant > DVCS communities and push them to request it. Where ?push? must be strictly limited by a continual awareness that the whole idea could just be bad. If you find yourself in a tiny minority pushing for a change, it *could* be that you have a great idea and the vast majority don't realise it yet. But you must be realistic about the likelihood that the change is a very *bad* idea, and frequently evaluate it for signs of that. -- \ ?I used to think that the brain was the most wonderful organ in | `\ my body. Then I realized who was telling me this.? ?Emo Philips | _o__) | Ben Finney From ben+python at benfinney.id.au Wed Jan 20 06:34:07 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:34:07 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> Message-ID: <87eillfigg.fsf@benfinney.id.au> "David Lyon" writes: > Well using that logic, distutils is an application.. Distutils is an application, the function of which is essential to allowing sane development of Python packages. It's a special case. We need to strictly limit the number of special cases, not gleefully add to them. -- \ ?I'd take the awe of understanding over the awe of ignorance | `\ any day.? ?Douglas Adams | _o__) | Ben Finney From matthieu.brucher at gmail.com Wed Jan 20 07:49:36 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 07:49:36 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. I think there are several points that make them not includable in Python: - git is not written in Python - bzr and mercurial have a life cycle much shorter than Python's, it's the same issue than with other libraries where another community develops them. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From david.lyon at pythontest.org Wed Jan 20 08:20:02 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 18:20:02 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Matthieu, >> I'd be happy to see bzr and mercurial and git all made it together >> into the stdlib for python 3. That would give a superb updating >> mechanism for python that would propel python well beyond >> the dinosaur badlands of CPAN and other languages. > > I think there are several points that make them not includable in Python: > - git is not written in Python > - bzr and mercurial have a life cycle much shorter than Python's, it's > the same issue than with other libraries where another community > develops them. That's only two points. :-) On 1; If that's true, I won't mention git again. On 2; Who knows what their life cycle is. CVS is pretty much dead, and svn looks like it is on the way out. I can't think of how anything could be better than mercurial or bzr but I know I will be proved wrong. At the end of the day, we are making a decision about whether the language is 'set-in-stone' or whether it is still evolving. To me, Python 1.x had it's own distinct "era", as has Python 2.x Hoping that the Python 3 era can be a little more flexible and perphaps "cleaner" than the 2.x era is all that I am thinking here. Have a nice day David From matthieu.brucher at gmail.com Wed Jan 20 09:05:19 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 09:05:19 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: > That's only two points. :-) In French, we say that several starts with 2 ;) > On 1; If that's true, I won't mention git again. I tis, you can check on the git repository (it's a mix of C, perl, shell scripts, Python, ...) > On 2; Who knows what their life cycle is. You can check on their websites, their cycles are far shorter than Python minor releases (several months vs several years). Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From stephen at xemacs.org Wed Jan 20 11:22:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 20 Jan 2010 19:22:55 +0900 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <87aaw9gjnk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > (Besides, git in the stdlib doesn't make much sense :). "Dulwich." From solipsis at pitrou.net Wed Jan 20 12:28:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Jan 2010 11:28:16 +0000 (UTC) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon pythontest.org> writes: > > I think I'll have better chances with PEPs. > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > [snip] This is python-ideas material, can you take it there? Thank you. Antoine. From ncoghlan at gmail.com Wed Jan 20 13:16:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 22:16:34 +1000 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B56F422.5010607@gmail.com> David Lyon wrote: > On 2; Who knows what their life cycle is. CVS is pretty much > dead, and svn looks like it is on the way out. > I can't think of how anything could be better than > mercurial or bzr but I know I will be proved wrong. I believe you misunderstood what Matthieu meant by life cycle there: think "release cycle". If a project pushes out new releases significantly more often than every 18-24 months (as is currently true for all of the major SCM tools), then that fact alone makes it a very bad fit for the Python standard library. And centralised source control will be going strong for years. The DVCS approach may be great for the open source world, but the gains are far more limited in a closed source shop (especially a group writing internal corporate applications which doesn't need to keep many, if any, maintenance branches going). If we weren't dealing with 4 active branches, the DVCS discussion would have got a lot less traction with the core developers - aside from better handling of multiple lines of development, most of the benefits of the switch to a DVCS accrue to people without commit access to the SVN repository. Anyway, we've wandered far afield from legit python-dev topics now. Any further ideas about super_mega_easy_install functionality that can pull code from source control systems and build it rather than requiring prebuild source tarballs should be directed to python-ideas (they probably need to bake more even before they make an appearance on distutils-sig). Cheers, Nick. P.S. As Jesse said... your enthusiasm is great, but please don't assume that some inherent conservatism on the part of other developers is automatically evil or the result of a failure to see your point. A lot of people around the world rely on our stuff every day. We owe it to them to be measured in our actions and to put serious thought into any major changes or additions we make to the language and the standard library. For the current stage of its development, Python 3 is in a good place from our point of view - its major carrot has really always been the better Unicode support it offers, and the ever-increasing globalisation of the web will create more and more pressure pushing developers in that direction as the years go by. Sure, Python 3 cleans up assorted other things as well, but the change to the text processing model is the big one that is fundamentally incompatible with the architecture of the 2.x series. Compared to that change, everything else is just tinkering. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Fri Jan 1 16:07:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:07:20 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04D81D51@TK5EX14MBXC118.redmond.corp.microsoft.com> <94bdd2610912141458m52da21ear4970506a37214e6@mail.gmail.com> <4dab5f760912230700l38a597f7l855bb2304025d6d5@mail.gmail.com> Message-ID: <94bdd2611001010707h4043ff64x7476049e435852b@mail.gmail.com> 2009/12/23 Glyph Lefkowitz : [..] > 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". > > The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. ?In other words it's got "bin", "lib", "share", "etc", etc.. ?~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation. Right, good idea Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Fri Jan 1 16:32:27 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 1 Jan 2010 16:32:27 +0100 Subject: [Python-Dev] First draft of "sysconfig" In-Reply-To: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> References: <94bdd2610912121202l48d39325q6f4cdcd73f972d5c@mail.gmail.com> Message-ID: <94bdd2611001010732o38a563bcu6d0cc9122ed5c88f@mail.gmail.com> On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziad? wrote: > Hello, > > A while ago I've proposed to refactor the APIs that provides access to > the installation paths and configuration variables at runtime into a > single module called "sysconfig", and make it easier for all > implementations to work with them. I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek From status at bugs.python.org Fri Jan 1 18:07:11 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Jan 2010 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100101170711.A5AAF78654@psf.upfronthosting.co.za> ACTIVITY SUMMARY (12/25/09 - 01/01/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2537 open (+22) / 16902 closed (+19) / 19439 total (+41) Open issues with patches: 1016 Average duration of open issues: 705 days. Median duration of open issues: 462 days. Open Issues Breakdown open 2504 (+22) pending 32 ( +0) Issues Created Or Reopened (42) _______________________________ doc: Code is not always colored 12/30/09 CLOSED http://bugs.python.org/issue7487 reopened flox documention buglet for PyBuffer 12/26/09 CLOSED http://bugs.python.org/issue7577 created ronaldoussoren easy Behavior of operations on a closed file object is not documented 12/26/09 http://bugs.python.org/issue7578 created blep Patch to add docstrings to msvcrt 12/26/09 http://bugs.python.org/issue7579 created brian.curtin patch distutils makefile from python.org installer doesn't work on OS 12/27/09 http://bugs.python.org/issue7580 created bskaplan incomplete doc of zlib 12/27/09 CLOSED http://bugs.python.org/issue7581 created coolwanglu [patch] diff.py to use iso timestamp 12/27/09 http://bugs.python.org/issue7582 created techtonik patch doctest should normalize tabs when comparing output 12/27/09 http://bugs.python.org/issue7583 created techtonik datetime.rfcformat() for Date and Time on the Internet 12/27/09 http://bugs.python.org/issue7584 created techtonik [patch] difflib should separate filename from timestamp with tab 12/27/09 http://bugs.python.org/issue7585 created techtonik patch Typo in collections documentation 12/28/09 CLOSED http://bugs.python.org/issue7586 created nneonneo Python 3.1.1 source build issues 12/28/09 CLOSED http://bugs.python.org/issue7587 created sharmaag unittest.TestCase.shortDescription isn't short anymore 12/28/09 http://bugs.python.org/issue7588 created exarkun setup.py shouldn't try to build nis module when nis headers aren 12/28/09 CLOSED http://bugs.python.org/issue7589 created Arfrever patch 'exceptions' module mentioned in documentation 12/28/09 CLOSED http://bugs.python.org/issue7590 created July test_get_platform fails on 3.1 12/28/09 http://bugs.python.org/issue7591 created pitrou buildbot ssl module documentation: SSLSocket.unwrap description shown twi 12/28/09 http://bugs.python.org/issue7592 created mnewman Computed-goto patch for RE engine 12/29/09 http://bugs.python.org/issue7593 created akuchling patch, needs review shlex refactoring 12/29/09 http://bugs.python.org/issue7594 created ferringb patch, needs review Doc typo for select.kevent() 12/29/09 CLOSED http://bugs.python.org/issue7595 created whit537 test_logging sometimes fails 12/29/09 CLOSED http://bugs.python.org/issue7596 created pitrou curses.use_env implementation error 12/30/09 http://bugs.python.org/issue7597 created kanru patch Cannot install, problems with assembly? 12/30/09 CLOSED http://bugs.python.org/issue7598 created sh.00 (c)ElementTree can produce invalid XML 12/30/09 http://bugs.python.org/issue7599 created jwilk interpreter crashes before start 12/30/09 CLOSED http://bugs.python.org/issue7600 created mhh91 Installation - 2 tests failed: test_cmd_line test_xmlrpc 12/30/09 CLOSED http://bugs.python.org/issue7601 created sadhak Doc: make clean and make update do not delete or update Doc/tool 12/30/09 CLOSED http://bugs.python.org/issue7602 created flox There is no 'seq' type 12/30/09 CLOSED http://bugs.python.org/issue7603 created tom66 delattr __slots__ inconsistancy 12/30/09 CLOSED http://bugs.python.org/issue7604 created ferringb test_cmd_line fails with non-ascii path 12/30/09 http://bugs.python.org/issue7605 created pitrou test_xmlrpc fails with non-ascii path 12/30/09 http://bugs.python.org/issue7606 created pitrou stringlib fastsearch could be improved on 64-bit builds 12/30/09 http://bugs.python.org/issue7607 created pitrou PyUnicode_FromFormatV handles %R and %S incorrectly. 12/30/09 CLOSED http://bugs.python.org/issue7608 created alexandre.vassalotti Add --with-system-expat option 12/31/09 CLOSED http://bugs.python.org/issue7609 created Arfrever patch Cannot use both read and readline method in same ZipExtFile obje 12/31/09 http://bugs.python.org/issue7610 created lucifer shlex not posix compliant when parsing "foo#bar" 12/31/09 http://bugs.python.org/issue7611 created jjdmol2 Fix "pass and object" typo in Library Reference / Built-in Types 12/31/09 CLOSED http://bugs.python.org/issue7612 created ygale patch [cppcheck] found a regression : Invalid number of character ((). 12/31/09 CLOSED http://bugs.python.org/issue7613 created ettl.martin patch Python 2.6.4 segfaults 12/31/09 CLOSED http://bugs.python.org/issue7614 created ttsiod unicode_escape codec does not escape quotes 01/01/10 http://bugs.python.org/issue7615 created rhansen patch test_memoryview test_setitem_writable failures with Intel ICC 01/01/10 http://bugs.python.org/issue7616 created ivank distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option 01/01/10 http://bugs.python.org/issue7617 created Arfrever patch Issues Now Closed (46) ______________________ True division of integers could be more accurate 715 days http://bugs.python.org/issue1811 mark.dickinson patch API to clear most free lists 690 days http://bugs.python.org/issue2019 amaury.forgeotdarc patch unit test UnicodeWarning 687 days http://bugs.python.org/issue2100 ezio.melotti test_disutils fails 568 days http://bugs.python.org/issue3054 ronaldoussoren patch test_formatdate_usegmt fails on non-englisch locale 451 days http://bugs.python.org/issue4046 r.david.murray "??" in u"foo" raises a misleading error 410 days http://bugs.python.org/issue4328 r.david.murray zipfile - add __exit__ attribute to make ZipFile object compatib 287 days http://bugs.python.org/issue5511 ezio.melotti patch test_urllib2 fails - urlopen error file not on local host 271 days http://bugs.python.org/issue5625 orsenthil Improve --with-dbmliborder option 170 days http://bugs.python.org/issue6491 benjamin.peterson patch Move the special-case for integer objects out of PyBytes_FromObj 141 days http://bugs.python.org/issue6687 alexandre.vassalotti patch, 26backport setup.py fails to find headers of system libffi 104 days http://bugs.python.org/issue6943 benjamin.peterson patch, easy The replacement suggested for callable(x) in py3k is not equival 92 days http://bugs.python.org/issue7006 benjamin.peterson patch C/API - Document exceptions 89 days http://bugs.python.org/issue7033 lekma patch subprocess.check_output: "docstring has inconsistent leading whi 35 days http://bugs.python.org/issue7381 georg.brandl patch optparse Documentation References Example Files that do not Exis 30 days http://bugs.python.org/issue7404 georg.brandl patch datetime.datetime.isoformat truncation problem 29 days http://bugs.python.org/issue7413 amaury.forgeotdarc patch, needs review doc: Code is not always colored 0 days http://bugs.python.org/issue7487 georg.brandl float('nan')**2 != nan. float('nan')**2 error 33 on windows 13 days http://bugs.python.org/issue7534 mark.dickinson patch If a generator raises TypeError when being unpacked, an unrelate 10 days http://bugs.python.org/issue7548 amaury.forgeotdarc ctypes doc improvement: c_char_p 6 days http://bugs.python.org/issue7569 georg.brandl Strange issue : cursor.commit() with sqlite 5 days http://bugs.python.org/issue7572 ghaering tes_math fails Mac OS X 10.4 due to OverflowError in test_mtestf 5 days http://bugs.python.org/issue7575 mark.dickinson patch documention buglet for PyBuffer 2 days http://bugs.python.org/issue7577 georg.brandl easy incomplete doc of zlib 0 days http://bugs.python.org/issue7581 amaury.forgeotdarc Typo in collections documentation 0 days http://bugs.python.org/issue7586 georg.brandl Python 3.1.1 source build issues 0 days http://bugs.python.org/issue7587 amaury.forgeotdarc setup.py shouldn't try to build nis module when nis headers aren 1 days http://bugs.python.org/issue7589 benjamin.peterson patch 'exceptions' module mentioned in documentation 1 days http://bugs.python.org/issue7590 georg.brandl Doc typo for select.kevent() 0 days http://bugs.python.org/issue7595 georg.brandl test_logging sometimes fails 1 days http://bugs.python.org/issue7596 ezio.melotti Cannot install, problems with assembly? 0 days http://bugs.python.org/issue7598 ezio.melotti interpreter crashes before start 0 days http://bugs.python.org/issue7600 r.david.murray Installation - 2 tests failed: test_cmd_line test_xmlrpc 1 days http://bugs.python.org/issue7601 ezio.melotti Doc: make clean and make update do not delete or update Doc/tool 0 days http://bugs.python.org/issue7602 georg.brandl There is no 'seq' type 0 days http://bugs.python.org/issue7603 benjamin.peterson delattr __slots__ inconsistancy 0 days http://bugs.python.org/issue7604 benjamin.peterson PyUnicode_FromFormatV handles %R and %S incorrectly. 0 days http://bugs.python.org/issue7608 alexandre.vassalotti Add --with-system-expat option 1 days http://bugs.python.org/issue7609 benjamin.peterson patch Fix "pass and object" typo in Library Reference / Built-in Types 0 days http://bugs.python.org/issue7612 ezio.melotti patch [cppcheck] found a regression : Invalid number of character ((). 0 days http://bugs.python.org/issue7613 ezio.melotti patch Python 2.6.4 segfaults 0 days http://bugs.python.org/issue7614 mark.dickinson Fwd: Debian Bug#42318: urllib.py has problems with malformed pro 3436 days http://bugs.python.org/issue210849 shinnosuke urllib / urllib2 should cache 301 redirections 2425 days http://bugs.python.org/issue735515 pitrou fast modular exponentiation 2084 days http://bugs.python.org/issue936813 mark.dickinson patch bdist_deb - Debian packager 316 days http://bugs.python.org/issue1054967 astraw patch Carbon.Scrap.PutScrapFlavor 987 days http://bugs.python.org/issue1700507 ronaldoussoren Top Issues Most Discussed (10) ______________________________ 11 Add Option to Bind to a Local IP Address in httplib.py 462 days open http://bugs.python.org/issue3972 8 fast modular exponentiation 2084 days closed http://bugs.python.org/issue936813 6 [patch] difflib should separate filename from timestamp with ta 5 days open http://bugs.python.org/issue7585 6 [patch] diff.py to use iso timestamp 5 days open http://bugs.python.org/issue7582 6 Implement fastsearch algorithm for rfind/rindex 24 days open http://bugs.python.org/issue7462 5 test_xmlrpc fails with non-ascii path 2 days open http://bugs.python.org/issue7606 5 test_logging sometimes fails 1 days closed http://bugs.python.org/issue7596 5 Patch to add docstrings to msvcrt 6 days open http://bugs.python.org/issue7579 5 Distutils-based installer does not detect 64bit versions of Pyt 126 days open http://bugs.python.org/issue6792 5 _sha256 et al. encode to UTF-8 by default 17 days open http://bugs.python.org/issue3745 From stefan_ml at behnel.de Sun Jan 3 09:11:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jan 2010 09:11:16 +0100 Subject: [Python-Dev] Providing support files to assist 3.x extension authors In-Reply-To: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> References: <99e0b9530912191638u757d2ce5mdf15394482cc19c0@mail.gmail.com> Message-ID: Case Vanhorsen, 20.12.2009 01:38: > When I ported gmpy (Python to GMP multiple precision library) to > Python 3.x, I began to use PyLong_AsLongAndOverflow frequently. I > found the code to slightly faster and cleaner than using PyLong_AsLong > and checking for overflow. You might want to look at the code Cython generates for integer type conversions. We use specialised coercion code that gets generated on-the-fly to convert Python long/int from and to all sorts of C integer types with compile time (portability) and runtime size/value checks. Depending on your needs, this may or may not be faster than the above C-API function. Stefan From david.lyon at preisshare.net Mon Jan 4 00:42:15 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 10:42:15 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> Message-ID: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> > On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: > > This new operator removes the ambiguity the original proposal had, > without making it more > complex for common use cases. So if you dislike it, you will need to > propose something > else that also fixes the ambiguity we had. Ok. > Environment markers >.. >Here are some example of fields using such markers: > >Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: [Windows] pywin32 1.0+ That's simpler, shorter, and less ambiguous. Easier to parse for package managers. David From python at mrabarnett.plus.com Mon Jan 4 01:14:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 04 Jan 2010 00:14:43 +0000 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4132F3.7020905@mrabarnett.plus.com> David Lyon wrote: >> On Mon, Dec 28, 2009 at 1:15 AM, Tarek Ziade wrote: >> >> This new operator removes the ambiguity the original proposal had, >> without making it more >> complex for common use cases. So if you dislike it, you will need to >> propose something >> else that also fixes the ambiguity we had. > > Ok. > >> Environment markers >> .. >> Here are some example of fields using such markers: >> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. > 'win32' is more specific than 'Windows' and, to me, '1.0+' means '>=1.0', not '>1.0'. From martin at v.loewis.de Mon Jan 4 01:21:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jan 2010 01:21:53 +0100 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4134A1.5090203@v.loewis.de> >> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' > > Requires-Dist: [Windows] pywin32 1.0+ > > That's simpler, shorter, and less ambiguous. Easier to > parse for package managers. Don't you want the PEP to complete? Why this bike-shedding? I can agree it's shorter. I can't agree that it's simpler, or less ambiguous. Regards, Martin From ssteinerx at gmail.com Mon Jan 4 01:29:14 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 3 Jan 2010 19:29:14 -0500 Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: On Jan 3, 2010, at 7:21 PM, Martin v. L?wis wrote: >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Really.... Enough, already! S From david.lyon at preisshare.net Mon Jan 4 01:47:56 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 4 Jan 2010 11:47:56 +1100 (EST) Subject: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2 In-Reply-To: <4B4134A1.5090203@v.loewis.de> References: <94bdd2610912211625k6a52dc19ue7dd7cad1e4f655c@mail.gmail.com> <75d5dd69b850e9bd793b43618327e655@preisshare.net> <871vilqhsy.fsf@uwakimon.sk.tsukuba.ac.jp> <4B3333DF.40705@v.loewis.de> <94bdd2610912240152r6131ec9ay2ada83f66aa17b35@mail.gmail.com> <4B334109.2060708@v.loewis.de> <4B346ACE.2030400@gmail.com> <94bdd2610912271601h69b081adsc871c849d1e2e1cc@mail.gmail.com> <4203.115.128.50.49.1261959349.squirrel@syd-srv02.ezyreg.com> <94bdd2610912271637m7b3df609m780622f74ee5daa@mail.gmail.com> <1255.218.214.45.58.1262562135.squirrel@syd-srv02.ezyreg.com> <4B4134A1.5090203@v.loewis.de> Message-ID: <1349.218.214.45.58.1262566076.squirrel@syd-srv02.ezyreg.com> Hi Martin, Happy New Year, >>> Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' >> >> Requires-Dist: [Windows] pywin32 1.0+ >> >> That's simpler, shorter, and less ambiguous. Easier to >> parse for package managers. > > Don't you want the PEP to complete? Why this bike-shedding? Well, I'm just helping out by pointing out some simpler ways as Tarek asked me. I was only answering his question. I was out celebrating so it took longer to reply than normal. Bike-Shedding ? Me ? which bikeshed? wanting simple? Anyway, I'm just reading the PEPs and commenting. If there are some alterations that can be done, lets discuss them. > I can agree it's shorter. > .. Cool. What I'd really like is a 'Code-Repository:' keyword so that we can install programs/libraries directly into a system. I feel that this would really simplify the packaging landscape, making it much easier for the scientific community and others to do python software installs. This would allow us to perphaps have something even *more modern* than CPAN. So yes, getting PEP-345 right is important to me. Have a nice day. David From abpillai at gmail.com Mon Jan 4 10:08:05 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 4 Jan 2010 14:38:05 +0530 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: <8548c5f31001040108v635a78ech33521371c6c50e82@mail.gmail.com> On Sun, Dec 27, 2009 at 11:21 PM, anatoly techtonik wrote: > On Tue, Dec 15, 2009 at 12:37 AM, Steven Bethard > wrote: > > > > If you're only concerned about 2.X, then yes, optparse will *never* be > > removed from 2.X. There will be a deprecation note in the 2.X > > documentation but deprecation warnings will only be issued when the -3 > > flag is specified. Please see the "Deprecation of optparse" section of > > the PEP: > > > > http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Ref http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse . Considering that optparse will be deprecated like 5 years from now. I think this point is moot. The deprecation strategy IMHO is perhaps way too conservative. Maybe it should be deprecated faster ;) -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From fetchinson at googlemail.com Mon Jan 4 10:32:10 2010 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 4 Jan 2010 10:32:10 +0100 Subject: [Python-Dev] Pronouncement on PEP 389: argparse? In-Reply-To: References: <24ea26600912141112i31a9eba7j7d06837456508094@mail.gmail.com> <4B26A41F.5020009@gmail.com> <24ea26600912141310w77d07c42hc244f9ecc1642b9f@mail.gmail.com> Message-ID: >> If you're only concerned about 2.X, then yes, optparse will *never* be >> removed from 2.X. There will be a deprecation note in the 2.X >> documentation but deprecation warnings will only be issued when the -3 >> flag is specified. Please see the "Deprecation of optparse" section of >> the PEP: >> >> http://www.python.org/dev/peps/pep-0389/#deprecation-of-optparse > > I do not think that optparse should be deprecated at. It is good at > what it does and its limitations make its starting point less > confusing for people with different backgrounds that Python. > > Compare: > http://docs.python.org/library/optparse.html > http://argparse.googlecode.com/svn/tags/r101/doc/index.html > > argparse should be recommended as advanced and more featured variant > of optparse - that goes without saying, but forcing people to switch > and annoying them with deprecation messages brings only headache. As has been noted already nobody is forcing people to switch. Optparse will be available as a separate package and everybody will be free to install it and will not have any deprecation messages anywhere. > Just > like optparse is better getopt, the latter could also be useful for > people coming from other languages and porting their libraries to > Python. > > I would better concentrate on real code examples how argparse solves > problems and would really want to see > http://www.python.org/dev/peps/pep-0389/#out-of-scope-various-feature-requests > implemented until argparse enters phase where backward incompatible > changes in API are impossible. > > I would prefer to see PEP 389 as a document describing proposed > solutions to argument parsing problems rather than petition to replace > one library with another. So, it should display common argument > parsing scenarios (use cases) with examples that are also useful > recipes for documentation. I guess more than 90% people here doesn't > have time to read argparse methods descriptions to see what they could > be useful for. -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From olemis at gmail.com Mon Jan 4 15:24:05 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 4 Jan 2010 09:24:05 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> Message-ID: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: > Thanks for the quick response. > > On 30/12/2009, Benjamin Peterson wrote: >> >> When I made that change, I didn't know that the __unittest "hack" was >> being used elsewhere outside of unittest, so I felt fine replacing it >> with another. While I still consider it an implementation detail, I >> would be ok with exposing an "official" API for this. Perhaps >> __unittest_ignore_traceback? > > Well, bazaar has had the trick for a couple of years, and googling > around now turns up some other projects using it or thinking about it: > > > > Add `dutest` and probably `nose` [2]_ and ... > Reinstating the old implementation (with the same name) would mean > that existing code would keep working with Python 2.7 IOW ... if it is removed all the libraries will have to change and check for the Py version and ... (probably not a big deal depending on the details of the ?solution?) > but maybe a > discussion could start about a new, less hacky, way of doing the same > I am strongly -1 for modifying the classes in ?traditional? unittest module [2]_ (except that I am strongly +1 for the package structure WITHOUT TOUCHING anything else ...) , and the more I think about it I am more convinced ... but anyway, this not a big deal (and in the end what I think is not that relevant either ... :o). So ... pass > May not be worthwhile making life more complicated though, > there aren't *that* many unittest-extending projects. > But every library extending `unittest` will do it or otherwise not-so-useful error messages will be displayed ;o) PS: Happy New Year ! BTW .. [1] [feature] nosexml should omit trace frames where `__unittest... (http://code.google.com/p/python-nosexml/issues/detail?id=4#c0) .. [2] Assessment of unittest 2.7 API : new features and opinions (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free new ripit 1.3.3 Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/KFwyUTKH0vI/ From juanfhj at gmail.com Tue Jan 5 17:10:16 2010 From: juanfhj at gmail.com (Juan Fernando Herrera J.) Date: Tue, 5 Jan 2010 11:10:16 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility Message-ID: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Jan 5 17:50:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 05 Jan 2010 16:50:15 +0000 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B436DC7.8040605@voidspace.org.uk> On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way > major software hasn't been ported to it. I'm eager to use 3, but > paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me > if this has been suggested in the past. > I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well. I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter). All the best, Michael Foord > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Tue Jan 5 18:21:31 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 5 Jan 2010 11:21:31 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion. I'm sure everyone out there would like to dive into supporting 3.x, but it's tough to prioritize when you are up to your eyeballs with 2.x bugs in your tracker. Look at Twisted ( http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3) -- over 1000 issues, ~5 developers -- 3.x support won't be here tomorrow, but it's on the horizon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 18:23:08 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2010 09:23:08 -0800 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B436DC7.8040605@voidspace.org.uk> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B436DC7.8040605@voidspace.org.uk> Message-ID: On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord wrote: > On 05/01/2010 16:10, Juan Fernando Herrera J. wrote: > > How about a new python 3 release with (possibly partial) backwards > compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major > software hasn't been ported to it. I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. > > > I don't know about other developers, but I certainly expected Python 3 > adoption to take a few years due to libraries only gradually making the > jump. I also expected there to be nervousness and pain around the migration > as well. > > I think in fact that libraries *are* migrating and there are lots of > encouraging signs. As a community we need to do all we can to promote Python > 3 adoption, but I don't think there is much reason to be worried (or > complacent for that matter). > Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Jan 5 18:52:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 18:52:20 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <319e029f1001050952o71cb29afh66445550beeb99c2@mail.gmail.com> On Tue, Jan 5, 2010 at 17:10, Juan Fernando Herrera J. wrote: > I'm eager to use 3, but paradoxically, > the 3 release makes me rather stuck with 2.6. Excuse me if this has been > suggested in the past. Yes. Python 3 is not what you want to use today if you write applications. If you write libraries 2010 is the year to port, IMO. With some luck, 2011 will be the year to start porting applications properly. We'll see. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ianb at colorstudy.com Tue Jan 5 20:00:49 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 13:00:49 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin wrote: > On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. wrote: > >> How about a new python 3 release with (possibly partial) backwards >> compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major >> software hasn't been ported to it. I'm eager to use 3, but paradoxically, >> the 3 release makes me rather stuck with 2.6. Excuse me if this has been >> suggested in the past. >> >> > The proper route to take, in my opinion, is to see what 2.x libraries you > are using that are not 3.x compatible, run 2to3 on them, then run their test > suite, and see where you get. Submit a patch or two to the library and see > what happens -- it at least gets the wheels in motion. > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 5 21:24:45 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 5 Jan 2010 15:24:45 -0500 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> On Jan 5, 2010, at 2:00 PM, Ian Bicking wrote: > It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. It seems like this is a problem to be addressed, then. Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. > The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). No-op constructions like 'bytes("")' could help for older versions of Python, though. A very, very small runtime shim could provide support for these, if 2to3 could be told about it somehow. > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? Sort of like our own version of the #line directive? :) Seriously though, I find it hard to believe that this is a big problem. The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. > I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. This is almost certainly more of a pain than trying to trick 2to3 into doing the right thing. From regebro at gmail.com Tue Jan 5 21:57:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Jan 2010 21:57:35 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <319e029f1001051257k3827c52ahcd115b15d9a8ece7@mail.gmail.com> On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz wrote: > It seems like this is a problem to be addressed, then. ?Let's get the "black magic" to be better specified and documented. is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well. Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x. >> The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated. > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 5 22:07:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 22:07:23 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <4B43AA0B.5030308@v.loewis.de> > It's not even that easy -- libraries can't apply patches for Python 3 > compatibility as they usually break Python 2 compatibility. Potentially > libraries could apply patches that make a codebase 2to3 ready, but from > what I've seen that's more black magic than straight forward updating, > as such patches have to trick 2to3 producing the output that is desired. I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x. > The only workable workflow I've seen people propose for maintaining a > single codebase with compatibility across both 2 and 3 is to use such > tricks, with aliases to suppress some 2to3 updates when they are > inappropriate I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way. > Also, running 2to3 on installation is kind of annoying, as you get > source that isn't itself the canonical source, so to fix bugs you have > to look at the installed source and trace it back to the bug in the > original source. That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin From martin at v.loewis.de Tue Jan 5 22:13:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jan 2010 22:13:07 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <34F4E992-621D-4400-B7CF-865C38BB7755@twistedmatrix.com> Message-ID: <4B43AB63.3000607@v.loewis.de> > No-op constructions like 'bytes("")' could help for older versions of > Python, though. A very, very small runtime shim could provide > support for these, if 2to3 could be told about it somehow. You actually don't *need* 2to3 support for that - bytes("") can work in either version: 2.x: def bytes(s): return s 3.x: def bytes(s) return s.encode("ascii") On top of that, you *could* as for bytes("") to be converted to b"" in 3.x - and it is indeed possible to tell 2to3 about that, and you don't even need to modify 2to3's source to do so: it can be part of your package. >> Also, running 2to3 on installation is kind of annoying, as you get >> source that isn't itself the canonical source, so to fix bugs you >> have to look at the installed source and trace it back to the bug >> in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in > .pycs rather than based on where the code was actually loaded in the > filesystem, couldn't 2to3 could do .pyc rewriting to point at the > original source? Sort of like our own version of the #line > directive? :) I think it could, but it would be fairly flaky as the pycs can get lost, and lose that information after regeneration. Also, it may be confusing in other scenarios. I'd rather have it generate separate mapper files, and change the traceback printing to consider these (as an option). > Seriously though, I find it hard to believe that this is a big > problem. The 3.x source looks pretty similar to the 2.x source, and > it's good to look at both if you're dealing with a 3.x issue. That's my experience as well - the only challenge is that you can't cut-n-paste the source URL into an editor. Regards, Martin From ianb at colorstudy.com Tue Jan 5 22:39:21 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 5 Jan 2010 15:39:21 -0600 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43AA0B.5030308@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: On Tue, Jan 5, 2010 at 3:07 PM, "Martin v. L?wis" wrote: > > > It's not even that easy -- libraries can't apply patches for Python 3 > > compatibility as they usually break Python 2 compatibility. ?Potentially > > libraries could apply patches that make a codebase 2to3 ready, but from > > what I've seen that's more black magic than straight forward updating, > > as such patches have to trick 2to3 producing the output that is desired. > > I wouldn't qualify it in that way. It may be necessary occasionally to > trick 2to3, but that's really a bug in 2to3 which you should report, so > that trickery is then a work-around for a bug - something that you may > have to do with other API, as well. > > The "black magic" is really more in the parts that 2to3 doesn't touch > at all (because they are inherently not syntactic); these are the > problem areas Guido refers to. The "black magic" then is to make the > same code work unmodified for both 2.x and 3.x. Just to clarify, the black magic I'm referring to is things like: try: ?? ?unicode_ = unicode except NameError: ?? ?unicode_ = str and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). ?If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. ?It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. ?For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str. ?That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself. Also replying to Glyph: > > Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. > > Given the way tracebacks are built, i.e. from filenames stored in .pycs rather than based on where the code was actually loaded in the filesystem, couldn't 2to3 could do .pyc rewriting to point at the original source? ?Sort of like our own version of the #line directive? :) > > Seriously though, I find it hard to believe that this is a big problem. ?The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue. Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun. So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits. Which is a terribly crotchety of me. Sorry. -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From martin at v.loewis.de Tue Jan 5 23:23:53 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 05 Jan 2010 23:23:53 +0100 Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> Message-ID: <4B43BBF9.2000302@v.loewis.de> > Just to clarify, the black magic I'm referring to is things like: > > try: > unicode_ = unicode > except NameError: > unicode_ = str > > and some other aliases like this that are unambiguous and which 2to3 > won't touch (if you write them correctly). If the porting guide noted > all these tricks (of which several have been developed, and I'm only > vaguely aware of a few) that would be helpful. It's not a lot of > tricks, but the tricks are not obvious and 2to3 gets the translation > wrong pretty often without them. For instance, when I say str in > Python 2 I often means bytes, unsurprisingly, but 2to3 translates both > str and unicode to str. No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3? > That *nothing* translates to bytes by default > (AFAIK) means that people must either be living in a bytes-free world > (which sure, lots of code does) or they are using tricks not included > in 2to3 itself. By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3). > Since 2to3 maintains line numbers yes, it wouldn't be that bad. But > then I don't currently develop any code that is "installed", I only > develop code that is directly from a source code checkout, and where > the checkout is put on the path. I guess I could have something that > automatically builds the code on every edit, and that's not > infeasible. It's just not fun. Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it. > So long as I have to support Python 2 > (which is like forever) then adding Python 3 only makes development > that much more complicated and much less fun, with no concrete > benefits. I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin From ziade.tarek at gmail.com Wed Jan 6 01:08:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 01:08:34 +0100 Subject: [Python-Dev] PEP 386 and PEP 345 Message-ID: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Hi, I think we've reached a consensus on those two PEPs. Although, there's one last point that was forgotten in the discussions : I've introduced "rc" in the pre-releases markers, so PEP 386 is compatible with Python's own version scheme. "rc" comes right after "c" in the sorting. It's slightly redundant with the "c" marker but I don't think this really matters as long as consumers know how to order them (a < b < c < rc). I have also stated that "c" is the preferred marker for third party projects, from PEP 386 point of view. Is there anything else I can do to make those two PEPs accepted ? Regards Tarek -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Wed Jan 6 01:26:34 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:26:34 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: <4B43BBF9.2000302@v.loewis.de> References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> <4B43AA0B.5030308@v.loewis.de> <4B43BBF9.2000302@v.loewis.de> Message-ID: <1322.218.214.45.58.1262737594.squirrel@syd-srv02.ezyreg.com> Hi Martin, > ... but in the long run, starting now will have saved you a lot of > pressure in the long run, and stop users from switching away from > your packages because of lack of Python 3 support. In a production situation it works the other way around. If there's an application that requires twisted (or whatever package) then most people would use the appropriate interpreter to match the library. Since you guys all did your jobs so well :-) doing so is painless. Because there is so much "comfort" with the existing situation it makes it an effort for people to move to a different lounge chair. Namely python 3. I'd suggest that moving the package set (pypi) to python 3 could be kicked along with the help of some automated tools. I don't know what tools you guys have got. But I am very sure that if code analysis was provided to package developers on python 3 (so they don't have to run it themselves), then it would be like an even bigger tv screen in a bigger lounge room and it would assist in drawing them over. David From david.lyon at preisshare.net Wed Jan 6 01:36:24 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 11:36:24 +1100 (EST) Subject: [Python-Dev] Suggestion: new 3 release with backwards compatibility In-Reply-To: References: <1fb8b0211001050810q4d054a10v23593ab83368c3b4@mail.gmail.com> Message-ID: <1337.218.214.45.58.1262738184.squirrel@syd-srv02.ezyreg.com> Hi Ian, > The only workable workflow I've seen people propose for maintaining a > single > codebase with compatibility across both 2 and 3 is to use such tricks, > with > aliases to suppress some 2to3 updates when they are inappropriate, so that > you can run 2to3 on install and have a single canonical Python 2 source. > Python 2.7 won't help much (even though it is trying) as the introduction > of non-ambiguous constructions like b"" aren't compatible with previous > versions of Python and so can't be used in many libraries (support at > least > back to Python 2.5 is the norm for most libraries, I think). > > Also, running 2to3 on installation is kind of annoying, as you get source > that isn't itself the canonical source, so to fix bugs you have to look at > the installed source and trace it back to the bug in the original source. > > I suspect a reasonable workflow might be possible with hg and maybe patch > queues, but I don't feel familiar enough with those tools to map that out. That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David From david.lyon at preisshare.net Wed Jan 6 03:01:22 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 6 Jan 2010 13:01:22 +1100 (EST) Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: <1617.218.214.45.58.1262743282.squirrel@syd-srv02.ezyreg.com> > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? Tarek, Given that I helped out so much last year with the PEP in discussing different options, even if they weren't accepted, I really feel that it is unfair if my name isn't mentioned. It was a huge time sacrifice on my part. For example, even if I only managed to explain the version numbering and clarify how that worked. It did take me some time to do that. What I did do however, was spend a lot of time with the multiplatform "Markers". I still think that two short weeks more of "discussion" could resolve some issues. That discussion went for 4 months on distutils-ml. Look, major issues aside, can you make the following concessions on PEP-345 which I only feel will help it, namely: 1) Source-Repository: specify a code repository to install from 2) Streamline Requires-Python: by implementing "Markers" as noted by the PEP. A marker being something like "Requires-Python(windows): lxml". Otherwise remove the word marker from the PEP and just replace with "metacode". Markers are what were discussed on distutils-ml. Metacode is what is described in the PEP. 3) Remove the inconsistency and platform ambiguities. Especially for windows users. For example, "win32" is extremely confusing for windows users right now. As more and more systems now are 64 bit. Use the platform module, instead of the sys module for constants. I'll post to distutils-ml on this. I am certainly not trying to hold this PEP up, and I apologise on my part for my late attention. I will post to distutils-ml on these and i promise to keep my comments unheated and unwitty. Having said that, PEP-345 is *super-important*. A week or two or three more discussion and the issues can be resolved. We all just want to focus on being productive. It would be a great accomplishment for you to get PEP-345 approved and likewise for me getting mentioned even in a minor role as helping out on a PEP. So I'm hoping that you can make a few last minute concessions meaning that we can all happily go on our way in 2010. Regards David From brett at python.org Wed Jan 6 06:20:30 2010 From: brett at python.org (Brett Cannon) Date: Tue, 5 Jan 2010 21:20:30 -0800 Subject: [Python-Dev] PEP 386 and PEP 345 In-Reply-To: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> References: <94bdd2611001051608p421d73bjbaa5c4f831efac5c@mail.gmail.com> Message-ID: On Tue, Jan 5, 2010 at 16:08, Tarek Ziad? wrote: > Hi, > > I think we've reached a consensus on those two PEPs. > > Although, there's one last point that was forgotten in the discussions > : I've introduced "rc" in the pre-releases markers, so PEP 386 is > compatible with Python's own version scheme. "rc" comes right after > "c" in the sorting. It's slightly redundant with the "c" marker but I > don't think this really matters as long as consumers know how to order > them (a < b < c < rc). I have also stated that "c" is the preferred > marker for third party projects, from PEP 386 point of view. > > Is there anything else I can do to make those two PEPs accepted ? > As you said, consensus has been reached, so just Guido's BDFL stamp of approval is all I can think of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Jan 6 12:19:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:19:45 +0000 Subject: [Python-Dev] bug triage Message-ID: <4B4471D1.9020707@simplistix.co.uk> Hi All, Is there a high volume of incoming bugs to the Python tracker? If so, I'd like to help with triaging. I think I have all the necessary access, what I'm missing is the knowledge of how to set myself up to get notifications of new bugs... How do I do that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at voidspace.org.uk Wed Jan 6 12:24:37 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 06 Jan 2010 11:24:37 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: <4B4472F5.8000709@voidspace.org.uk> On 06/01/2010 11:19, Chris Withers wrote: > Hi All, > > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the > necessary access, what I'm missing is the knowledge of how to set > myself up to get notifications of new bugs... > > How do I do that? > Bug triaging is one of Python's "big needs" and anything you do to help on this score would be much appreciated. Particularly reviewing new and outstanding issues. I assumed there would be RSS feeds for bug tracker activity but can't easily find these on the tracker. There is a bot that posts activity to #python-dev, so there must be some way of getting this information. All the best, Michael > cheers, > > Chris > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Jan 6 12:25:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 11:25:16 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: Hi Chris, > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Do you really want to get such notifications? There may be a lot of them. If you want however, you can join #python-dev on IRC (irc.freenode.net) where there's a bot which posts updates of all bugs on the tracker. There's usually not a lot of discussion going on so you probably won't feel flooded. In addition to bug triage, what is needed is reviewing of existing patches, as well as writing patches for issues which haven't been addressed yet :-) Regards Antoine. From chris at simplistix.co.uk Wed Jan 6 12:30:40 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:30:40 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B447460.7080100@simplistix.co.uk> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. Yeah, email-out is what I'm really after... I have it for my own Roundup instance so it can't be that hard to do ;-) Roch?, you guys host the bug tracker, right? Is there email-out set up for it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Jan 6 12:37:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 21:37:23 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <4B4475F3.5040406@gmail.com> Michael Foord wrote: > I assumed there would be RSS feeds for bug tracker activity but can't > easily find these on the tracker. There is a bot that posts activity to > #python-dev, so there must be some way of getting this information. I'm pretty sure the bugs list is still the primary spooled notification mechanism: http://mail.python.org/mailman/listinfo/python-bugs-list There are also the weekly tracker activity summaries that are posted here to python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From chris at simplistix.co.uk Wed Jan 6 12:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 11:41:28 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <4B4476E8.8050709@simplistix.co.uk> Nick Coghlan wrote: > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list That's what I was after, thanks! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From facundobatista at gmail.com Wed Jan 6 13:03:08 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 6 Jan 2010 09:03:08 -0300 Subject: [Python-Dev] bug triage In-Reply-To: <4B4471D1.9020707@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> Message-ID: On Wed, Jan 6, 2010 at 8:19 AM, Chris Withers wrote: > Is there a high volume of incoming bugs to the Python tracker? > If so, I'd like to help with triaging. I think I have all the necessary > access, what I'm missing is the knowledge of how to set myself up to get > notifications of new bugs... Not notifications, but maybe a way to have a higher look of them for easy selection: http://www.taniquetil.com.ar/cgi-bin/pytickets.py Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From ziade.tarek at gmail.com Wed Jan 6 13:29:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:29:58 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B4472F5.8000709@voidspace.org.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> Message-ID: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord wrote: > On 06/01/2010 11:19, Chris Withers wrote: >> >> Hi All, >> >> Is there a high volume of incoming bugs to the Python tracker? >> If so, I'd like to help with triaging. I think I have all the necessary >> access, what I'm missing is the knowledge of how to set myself up to get >> notifications of new bugs... >> >> How do I do that? >> > > Bug triaging is one of Python's "big needs" and anything you do to help on > this score would be much appreciated. Particularly reviewing new and > outstanding issues. Another useful triage I think, is to review the oldest bugs (some of them are > 5 years) and remove the ones that are not relevant anymore, or duplicate with newer entries. Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Wed Jan 6 13:31:08 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 06 Jan 2010 12:31:08 +0000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B44828C.4070201@simplistix.co.uk> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with someone as a paired task for those 2 days... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Jan 6 13:37:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 6 Jan 2010 13:37:55 +0100 Subject: [Python-Dev] bug triage In-Reply-To: <4B44828C.4070201@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B44828C.4070201@simplistix.co.uk> Message-ID: <94bdd2611001060437s2d0242aembc669c3263773fea@mail.gmail.com> On Wed, Jan 6, 2010 at 1:31 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Another useful triage I think, is to review the oldest bugs (some of >> them are > 5 years) >> and remove the ones that are not relevant anymore, or duplicate with >> newer entries. > > I'm sprinting for 2 days at PyCon, I'd verymuch be up for doing this with > someone as a paired task for those 2 days... I'll be doing Distutils stuff but I can probably help around a bit in that task: Distutils have quite a few old issues so I can tackle those From roche at upfrontsystems.co.za Wed Jan 6 13:32:39 2010 From: roche at upfrontsystems.co.za (=?ISO-8859-1?Q?Roch=E9?= Compaan) Date: Wed, 06 Jan 2010 14:32:39 +0200 Subject: [Python-Dev] bug triage In-Reply-To: <4B447460.7080100@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B447460.7080100@simplistix.co.uk> Message-ID: <1262781159.2836.218.camel@didi> On Wed, 2010-01-06 at 11:30 +0000, Chris Withers wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > Yeah, email-out is what I'm really after... I have it for my own Roundup > instance so it can't be that hard to do ;-) > > Roch?, you guys host the bug tracker, right? Is there email-out set up > for it? We do, but we don't administer it. There are a few administrators taking care of it and you should be able to reach them by logging your request here: http://psf.upfronthosting.co.za/roundup/meta/ or post it to the Infrastructure mailing list: infrastructure at python.org -- Roch? Compaan Upfront Systems http://www.upfrontsystems.co.za From ncoghlan at gmail.com Wed Jan 6 13:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Jan 2010 22:57:24 +1000 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <4B4488B4.2070208@gmail.com> Tarek Ziad? wrote: > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I believe someone (Daniel Diniz, maybe?) did do a pass over those some time in the last 12 months, so most of the obviously irrelevant ones that are that old should already be gone. Not to say it isn't worth doing another pass, just saying not to get disheartened if there aren't many that can be readily closed. There are at least a few still kicking around just because they're difficult to deal with (there's an ancient one to do with one of the ways circular imports can fail that I occasionally go back and reread before moving on to something more tractable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Wed Jan 6 14:43:17 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 08:43:17 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <4B4476E8.8050709@simplistix.co.uk> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> <4B4476E8.8050709@simplistix.co.uk> Message-ID: <20100106134317.3EF141FB5A6@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:41:28 +0000, Chris Withers wrote: > Nick Coghlan wrote: > > I'm pretty sure the bugs list is still the primary spooled notification > > mechanism: > > http://mail.python.org/mailman/listinfo/python-bugs-list > > That's what I was after, thanks! Just for completeness, there's also new-bugs-announce if you want just *new* bug notification. That's more for people who want to watch for bugs they want to become nosy on, though; if you are doing triage python-bugs-list is what you want. Please also read http://www.python.org/dev/workflow/ if you haven't already. Thanks for being willing to chip in! --David From brian.curtin at gmail.com Wed Jan 6 15:57:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 08:57:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4488B4.2070208@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > I believe someone (Daniel Diniz, maybe?) did do a pass over those some > time in the last 12 months, so most of the obviously irrelevant ones > that are that old should already be gone. Not to say it isn't worth > doing another pass, just saying not to get disheartened if there aren't > many that can be readily closed. > > There are at least a few still kicking around just because they're > difficult to deal with (there's an ancient one to do with one of the > ways circular imports can fail that I occasionally go back and reread > before moving on to something more tractable). > > Cheers, > Nick. > On the topic of bugs that can be readily closed (literally), I've recently come across a number of issues which appear to be sitting in a patch or review stage, but their patches have been committed and the issue remains open. What is the best course of action there? I'd just go ahead and close the issue myself but I don't have tracker privileges. I'm willing to help out with another Daniel Diniz-esque triage sweep if that would help. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Wed Jan 6 16:14:20 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 6 Jan 2010 10:14:20 -0500 Subject: [Python-Dev] bug triage In-Reply-To: <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> Message-ID: <7FCF8B19-3465-4392-80CC-BEAEBD926ECA@gmail.com> On Jan 6, 2010, at 7:29 AM, Tarek Ziad? wrote: > On Wed, Jan 6, 2010 at 12:24 PM, Michael Foord > wrote: >> On 06/01/2010 11:19, Chris Withers wrote: >>> >>> Hi All, >>> >>> Is there a high volume of incoming bugs to the Python tracker? >>> If so, I'd like to help with triaging. I think I have all the necessary >>> access, what I'm missing is the knowledge of how to set myself up to get >>> notifications of new bugs... >>> >>> How do I do that? >>> >> >> Bug triaging is one of Python's "big needs" and anything you do to help on >> this score would be much appreciated. Particularly reviewing new and >> outstanding issues. > > Another useful triage I think, is to review the oldest bugs (some of > them are > 5 years) > and remove the ones that are not relevant anymore, or duplicate with > newer entries. I was actually thinking about that the other day when I saw that the average age of bugs on the Python tracker was at some hideously large 3 digit number. The 'success' statistic would be to bring that down below, say, 100. S From skip at pobox.com Wed Jan 6 17:38:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 6 Jan 2010 10:38:13 -0600 Subject: [Python-Dev] bug triage In-Reply-To: <4B4475F3.5040406@gmail.com> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: <19268.48245.616046.874126@montanaro.dyndns.org> >>>>> "Nick" == Nick Coghlan writes: Nick> I'm pretty sure the bugs list is still the primary spooled Nick> notification mechanism: Nick> http://mail.python.org/mailman/listinfo/python-bugs-list Actually, there is a new-bugs-announce list: http://mail.python.org/mailman/listinfo/new-bugs-announce Skip From solipsis at pitrou.net Wed Jan 6 18:56:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 17:56:49 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > On the topic of bugs that can be readily closed (literally), I've > recently come across a number of issues which appear to be sitting in a > patch or review stage, but their patches have been committed and the > issue remains open. What is the best course of action there? Post a message on the issue asking for info. From solipsis at pitrou.net Wed Jan 6 19:09:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Jan 2010 18:09:39 +0000 (UTC) Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Le Wed, 06 Jan 2010 08:57:42 -0600, Brian Curtin a ?crit?: > > On the topic of bugs that can be readily closed (literally), I've > > recently come across a number of issues which appear to be sitting in a > > patch or review stage, but their patches have been committed and the > > issue remains open. What is the best course of action there? > > Post a message on the issue asking for info. Ok, I realize my answer might have been a bit terse :-) The patch might be waiting to be merged in all development branches, or it may not totally resolve the issue, or perhaps documentation needs to be updated, or perhaps it is pending a verdict from the buildbots, etc. You can't deduce that the issue is completely fixed from the simple fact that something has been committed. Regards Antoine Pitrou. From brett at python.org Wed Jan 6 20:03:32 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 11:03:32 -0800 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > On Wed, Jan 6, 2010 at 06:57, Nick Coghlan wrote: > >> I believe someone (Daniel Diniz, maybe?) did do a pass over those some >> time in the last 12 months, so most of the obviously irrelevant ones >> that are that old should already be gone. Not to say it isn't worth >> doing another pass, just saying not to get disheartened if there aren't >> many that can be readily closed. >> >> There are at least a few still kicking around just because they're >> difficult to deal with (there's an ancient one to do with one of the >> ways circular imports can fail that I occasionally go back and reread >> before moving on to something more tractable). >> >> Cheers, >> Nick. >> > > On the topic of bugs that can be readily closed (literally), I've recently > come across a number of issues which appear to be sitting in a patch or > review stage, but their patches have been committed and the issue remains > open. What is the best course of action there? I'd just go ahead and close > the issue myself but I don't have tracker privileges. > > If a core developer is willing to step forward and vouch for you to get tracker privileges then I will give them to you. We are trying to give out tracker privs w/ less time than required to get commit privileges. So as long as you have helped out on a few issues in a positive and correct way that should be enough to get one of the regulars who perform triage to notice. -Brett I'm willing to help out with another Daniel Diniz-esque triage sweep if that > would help. > > Brian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Wed Jan 6 21:41:05 2010 From: nad at acm.org (Ned Deily) Date: Wed, 06 Jan 2010 12:41:05 -0800 Subject: [Python-Dev] bug triage References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <4B4475F3.5040406@gmail.com> Message-ID: In article <4B4475F3.5040406 at gmail.com>, Nick Coghlan wrote: > Michael Foord wrote: > > I assumed there would be RSS feeds for bug tracker activity but can't > > easily find these on the tracker. There is a bot that posts activity to > > #python-dev, so there must be some way of getting this information. > > I'm pretty sure the bugs list is still the primary spooled notification > mechanism: > http://mail.python.org/mailman/listinfo/python-bugs-list Also, that mailing list (along with most python development related mailing lists) is mirrored at gmane.org which means it can also be obtained via a newsreader (NNTP) or various RSS feeds. http://dir.gmane.org/gmane.comp.python.bugs -- Ned Deily, nad at acm.org From nick.bastin at gmail.com Wed Jan 6 22:14:54 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:14:54 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? Message-ID: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> (This may occur on more platforms - I can test on more unix platforms if the consensus is this is an actual problem and I'm not just a nut) On freebsd5, if you do a simple ./configure --enable-shared in current (2.7) trunk, your python shared library will build properly, but all modules will fail to find the shared library and thus fail to build: gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.7/u1/Python/Python-2.7a1/Modules/_struct.o -L/u1/tmp/python2.7a1/lib -L/usr/local/lib -lpython2.7 -o build/lib.freebsd-5.3-RELEASE-i386-2.7/_struct.so /usr/bin/ld: cannot find -lpython2.7 building '_ctypes_test' extension ... This of course is because libpython2.7.so is in the current directory and not (yet) installed in /usr/local/lib. I've made a very simple fix for this problem that works, but at least to me smells a bit funny, which is to modify setup.py to add the following to detect_modules(): # If we did --enable-shared, we need to be able to find the library # we just built in order to build the modules. if platform == 'freebsd5': add_dir_to_list(self.compiler_obj.library_dirs, '.') Which brings me to a few questions: a) Does this seem like a real problem, or am I missing something obvious? b) Does this fix seem like the sensible thing to do? (it seems at least that we ought to check that the user configured --enable-shared and only set -L. in that case, if that's possible) Setting --enable-shared when you actually have a libpython2.7.so in /usr/local/lib (or whatever --prefix you've selected) is possibly even more dangerous, because it may succeed in linking against a differently-built library than what you intended. -- Nick From nick.bastin at gmail.com Wed Jan 6 22:39:17 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 16:39:17 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <66d0a6e11001061339t38202b70mca1091e1926ecf4b@mail.gmail.com> On Wed, Jan 6, 2010 at 16:14, Nicholas Bastin wrote: > This of course is because libpython2.7.so is in the current directory > and not (yet) installed in /usr/local/lib. One minor correction - as you could see from the compile line, the actual --prefix in this case is /u1/tmp/python2.7a1, but the libraries obviously aren't installed there yet either. Perhaps a better fix than setting -L. would be to put the shared library in build/lib.freebsd-5.3-RELEASE-i386-2.7 and add that to the library path for the linker (the build creates this directory, but installs nothing in it). -- Nick From martin at v.loewis.de Wed Jan 6 23:21:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jan 2010 23:21:44 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> Message-ID: <4B450CF8.8090009@v.loewis.de> > b) Does this fix seem like the sensible thing to do? No. Linking in setup.py should use the same options as if the module was built as *shared* through Modules/Setup, which, IIUC, should use BLDLIBRARY. Regards, Martin From nick.bastin at gmail.com Thu Jan 7 01:08:13 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Wed, 6 Jan 2010 19:08:13 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B450CF8.8090009@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> Message-ID: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> On Wed, Jan 6, 2010 at 17:21, "Martin v. L?wis" wrote: >> b) Does this fix seem like the sensible thing to do? > > No. Linking in setup.py should use the same options as if the module > was built as *shared* through Modules/Setup, which, IIUC, should use > BLDLIBRARY. Thanks for that pointer, that makes much more sense. Indeed, BLDLIBRARY on FreeBSD* is set to '-L. -lpython$(VERSION)' if you set --enable-shared, but somehow that piece of information doesn't propagate into the module build. More investigation to be done... -- Nick From rdmurray at bitdance.com Thu Jan 7 02:22:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 06 Jan 2010 20:22:42 -0500 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> Message-ID: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > On Wed, Jan 6, 2010 at 06:57, Brian Curtin wrote: > > On the topic of bugs that can be readily closed (literally), I've recently > > come across a number of issues which appear to be sitting in a patch or > > review stage, but their patches have been committed and the issue remains > > open. What is the best course of action there? I'd just go ahead and close > > the issue myself but I don't have tracker privileges. > > > > > If a core developer is willing to step forward and vouch for you to get > tracker privileges then I will give them to you. We are trying to give out > tracker privs w/ less time than required to get commit privileges. So as > long as you have helped out on a few issues in a positive and correct way > that should be enough to get one of the regulars who perform triage to > notice. > > -Brett I've done a quick scan of issues Brian is nosy on to refresh my memory, and I'd say he's definitely been making positive contributions. I'm willing to volunteer to keep an eye on his triage work for a while if you grant him tracker privs. Brian, I assume you'll be cognizant of Antoine's advice about making sure a bug really should be closed before closing it :) Hanging out in #python-dev on freenode while working on issues can be helpful, as well, since you can quickly ask whoever is there for second opinions on particular bugs. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Thu Jan 7 02:28:26 2010 From: brett at python.org (Brett Cannon) Date: Wed, 6 Jan 2010 17:28:26 -0800 Subject: [Python-Dev] bug triage In-Reply-To: <20100107012242.2C7D11FBB50@kimball.webabinitio.net> References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > > On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: > > On Wed, Jan 6, 2010 at 06:57, Brian Curtin > wrote: > > > On the topic of bugs that can be readily closed (literally), I've > recently > > > come across a number of issues which appear to be sitting in a patch or > > > review stage, but their patches have been committed and the issue > remains > > > open. What is the best course of action there? I'd just go ahead and > close > > > the issue myself but I don't have tracker privileges. > > > > > > > > If a core developer is willing to step forward and vouch for you to get > > tracker privileges then I will give them to you. We are trying to give > out > > tracker privs w/ less time than required to get commit privileges. So as > > long as you have helped out on a few issues in a positive and correct way > > that should be enough to get one of the regulars who perform triage to > > notice. > > > > -Brett > > I've done a quick scan of issues Brian is nosy on to refresh my > memory, and I'd say he's definitely been making positive contributions. > I'm willing to volunteer to keep an eye on his triage work for a while > if you grant him tracker privs. > > Done for the username brian.curtin (email doesn't match the one Brian emailed from so do let me know, Brian if this is the right username). Welcome aboard! -Brett > Brian, I assume you'll be cognizant of Antoine's advice about making > sure a bug really should be closed before closing it :) Hanging out in > #python-dev on freenode while working on issues can be helpful, as well, > since you can quickly ask whoever is there for second opinions on > particular bugs. > > -- > R. David Murray www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Jan 7 02:59:42 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 6 Jan 2010 19:59:42 -0600 Subject: [Python-Dev] bug triage In-Reply-To: References: <4B4471D1.9020707@simplistix.co.uk> <4B4472F5.8000709@voidspace.org.uk> <94bdd2611001060429q19287b06i51cbf51cfa43f582@mail.gmail.com> <4B4488B4.2070208@gmail.com> <20100107012242.2C7D11FBB50@kimball.webabinitio.net> Message-ID: On Wed, Jan 6, 2010 at 19:28, Brett Cannon wrote: > > > On Wed, Jan 6, 2010 at 17:22, R. David Murray wrote: > >> >> On Wed, 06 Jan 2010 11:03:32 -0800, Brett Cannon wrote: >> > On Wed, Jan 6, 2010 at 06:57, Brian Curtin >> wrote: >> > > On the topic of bugs that can be readily closed (literally), I've >> recently >> > > come across a number of issues which appear to be sitting in a patch >> or >> > > review stage, but their patches have been committed and the issue >> remains >> > > open. What is the best course of action there? I'd just go ahead and >> close >> > > the issue myself but I don't have tracker privileges. >> > > >> > > >> > If a core developer is willing to step forward and vouch for you to get >> > tracker privileges then I will give them to you. We are trying to give >> out >> > tracker privs w/ less time than required to get commit privileges. So as >> > long as you have helped out on a few issues in a positive and correct >> way >> > that should be enough to get one of the regulars who perform triage to >> > notice. >> > >> > -Brett >> >> I've done a quick scan of issues Brian is nosy on to refresh my >> memory, and I'd say he's definitely been making positive contributions. >> I'm willing to volunteer to keep an eye on his triage work for a while >> if you grant him tracker privs. >> >> > Done for the username brian.curtin (email doesn't match the one Brian > emailed from so do let me know, Brian if this is the right username). > Welcome aboard! > > Yep, that's the one. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Jan 7 04:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 07 Jan 2010 03:07:56 +0000 Subject: [Python-Dev] GIL required for _all_ Python calls? Message-ID: <4B45500C.8090905@mrabarnett.plus.com> Hi, I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? Or is it just a case of checking the source files for mentions of the GIL? The header file for PyList_New, for example, doesn't mention it! Thanks From john.arbash.meinel at gmail.com Thu Jan 7 04:25:48 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 06 Jan 2010 21:25:48 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B45543C.2090200@gmail.com> MRAB wrote: > Hi, > > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? Or is it just a case of checking the source > files for mentions of the GIL? The header file for PyList_New, for > example, doesn't mention it! > > Thanks Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may get concurrent updating of the value, and then the final value is wrong. (two threads do 5+1 getting 6, rather than 7, and when the decref, you end up at 4 rather than back at 5). AFAIK, the only things that don't require the GIL are macro functions, like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for example, will be increfing and setting the exception state, so certainly needs the GIL to be held. John =:-> From benjamin at python.org Thu Jan 7 04:32:17 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jan 2010 21:32:17 -0600 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45543C.2090200@gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> Message-ID: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> 2010/1/6 John Arbash Meinel : > Anything that Py_INCREF or Py_DECREF's should have the GIL, or you may > get concurrent updating of the value, and then the final value is wrong. > (two threads do 5+1 getting 6, rather than 7, and when the decref, you > end up at 4 rather than back at 5). Correct. > > AFAIK, the only things that don't require the GIL are macro functions, > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > example, will be increfing and setting the exception state, so certainly > needs the GIL to be held. As a general rule, I would say, no Py* macros are safe without the gil either (the exception being Py_END_ALLOW_THREADS), since they mutate Python objects which must be protected. -- Regards, Benjamin From guido at python.org Thu Jan 7 05:29:03 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Jan 2010 20:29:03 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: On Wed, Jan 6, 2010 at 7:32 PM, Benjamin Peterson wrote: > 2010/1/6 John Arbash Meinel : > > AFAIK, the only things that don't require the GIL are macro functions, > > like PyString_AS_STRING or PyTuple_SET_ITEM. PyErr_SetString, for > > example, will be increfing and setting the exception state, so certainly > > needs the GIL to be held. > > As a general rule, I would say, no Py* macros are safe without the gil > either (the exception being Py_END_ALLOW_THREADS), since they mutate > Python objects which must be protected. That's keeping it on the safe side, since there are some macros like PyString_AS_STRING() that are also safe, *if* you are owning at least one reference to the string object. At the same time, "no Py* macros" is not quite strong enough, since if you called PyString_AS_STRING() before releasing the GIL but you don't own a reference to the string object, the string might be deallocated behind your back by another thread. A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). -- --Guido van Rossum (python.org/~guido) From yoann.padioleau at facebook.com Thu Jan 7 08:24:42 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Wed, 6 Jan 2010 23:24:42 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt Message-ID: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a file called ast.txt. Where does this file come from ? Is it generated from Python.asdl ? From johan.gill at agama.tv Thu Jan 7 10:46:52 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 10:46:52 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. Message-ID: <4B45AD8C.5000405@agama.tv> Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before, so this is probably an already solved problem. Regards Johan Gill Agama Technologies From regebro at gmail.com Thu Jan 7 12:12:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 12:12:17 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45AD8C.5000405@agama.tv> References: <4B45AD8C.5000405@agama.tv> Message-ID: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I imagine that other companies have contributed before, so this is > probably an already solved problem. I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hodgestar+pythondev at gmail.com Thu Jan 7 12:28:01 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 7 Jan 2010 13:28:01 +0200 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon From swamy.sangamesh at gmail.com Thu Jan 7 11:46:59 2010 From: swamy.sangamesh at gmail.com (swamy sangamesh) Date: Thu, 7 Jan 2010 16:16:59 +0530 Subject: [Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9 Message-ID: Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have attached the full test case result. If i change the type c_int and c_short to c_unit and c_ushort of class "BITS(Structure)" in file test_bitfields.py then no failures are seen. Has anyone faced the similar issue or any help is appreciated. Thanks, Sangamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ctype-testcases Type: application/octet-stream Size: 22996 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 7 13:23:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 07 Jan 2010 22:23:11 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> Message-ID: <4B45D22F.40404@gmail.com> Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls we need to think >> about? I imagine that other companies have contributed before, so this is >> probably an already solved problem. > > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license without requesting additional permission from the copyright holder. For corporate contributions, I believe the contributor agreement needs to be signed by an authorised agent of the company - the place to check that would probably be psf at python.org (that's the email address for the PSF board). Assuming the subject line relates to the code that you would like to contribute though, that particular change is unlikely to happen - 2.6 is in maintenance mode and changing RLock from a Python implementation to the faster C one is solidly in new feature territory. Although a backport of the 3.2 C RLock implementation to 2.7 could be useful, I doubt that backporting code provided by an existing committer would be the subject of this query :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Thu Jan 7 14:11:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:11:27 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. Even if the contributed code as in this case is a method in an existing file? How does that work, how do they keep ownership of one method in the threading.py method? :-) > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) Ah. I probably misunderstood what the suggested contribution was. Maybe it was a separate file, which I didn't get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Thu Jan 7 14:15:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 07 Jan 2010 13:15:09 +0000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> Message-ID: <4B45DE5D.3030104@voidspace.org.uk> On 07/01/2010 13:11, Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > >> As Simon pointed out, while some organisations do work that way, the PSF >> isn't one of them. >> >> The PSF only requires that the code be contributed under a license that >> then allows us to turn around and redistribute it under a different open >> source license without requesting additional permission from the >> copyright holder. >> > Even if the contributed code as in this case is a method in an > existing file? How does that work, how do they keep ownership of one > method in the threading.py method? :-) > > When contributing code to Python all work remains copyright the original author. The combined work is copyright *everyone*. The PSF has a license to distribute it, which is all that is important. How you meaningfully exercise your ownership over chunks of code is left for the reader to determine... (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From stefan_ml at behnel.de Thu Jan 7 14:30:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:30:16 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: Guido van Rossum, 07.01.2010 05:29: > A better rule would be "you may access the memory buffer in a PyString > or PyUnicode object with the GIL released as long as you own a > reference to the string object." Everything else is out of bounds (or > not worth the bother). Is that a "yes" regarding the OP's original question about releasing the GIL during regexp searches? Stefan From regebro at gmail.com Thu Jan 7 14:36:42 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Jan 2010 14:36:42 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DE5D.3030104@voidspace.org.uk> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <319e029f1001070511i20d2aa4fw4a9e9b2f54fe84d8@mail.gmail.com> <4B45DE5D.3030104@voidspace.org.uk> Message-ID: <319e029f1001070536t49f76899j111212b02c14f192@mail.gmail.com> On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Thu Jan 7 15:05:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:05:23 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB, 07.01.2010 04:07: > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. > or PyErr_SetString? Certainly not safe. > Is there an easy way to find out? Release it and fix any crashes? Note that this isn't a safe solution, though, as some GIL requiring code may be platform specific. So a better approach might be to extract any obviously problematic stuff from the existing code (such as any exception handling, explicit ref-counting or object creation), and *then* try to release the GIL. Stefan From solipsis at pitrou.net Thu Jan 7 16:38:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 15:38:31 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or macros you want to call with the GIL released, and assess whether it is safe to call them. As already pointed out, no reference count should be changed, and generally no mutable container should be accessed, except if that container is known not to be referenced anywhere else (that would be the case for e.g. a list that your function has created and is busy populating). I agree that releasing the GIL when doing non-trivial regex searches is a worthwhile research, so please don't give up immediately :-) Regards Antoine Pitrou. From olemis at gmail.com Thu Jan 7 19:24:59 2010 From: olemis at gmail.com (Olemis Lang) Date: Thu, 7 Jan 2010 13:24:59 -0500 Subject: [Python-Dev] Question over splitting unittest into a package In-Reply-To: <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> References: <1afaf6160912301204p247e77c4r2271c4f37114ba4@mail.gmail.com> <24ea26601001040624wb7d44ffl6ca0190aa33f8553@mail.gmail.com> Message-ID: <24ea26601001071024m2abd988fuc77f94845d57483a@mail.gmail.com> On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am strongly -1 for modifying the classes in ?traditional? unittest > module [2]_ (except that I am strongly +1 for the package structure > WITHOUT TOUCHING anything else ...) , and the more I think about it I > am more convinced ... but anyway, this not a big deal (and in the end > what I think is not that relevant either ... :o). So ... > IOW, if I had all the freedom to implement it, after the pkg structure I'd do something like : {{{ #!python class TestResult: # Everything just the same def _is_relevant_tb_level(self, tb): return '__unittest' in tb.tb_frame.f_globals class BetterTestResult(TestResult): # Further code ... maybe ;o) # def _is_relevant_tb_level(self, tb): # This or anything else you might want to do ;o) # globs = tb.tb_frame.f_globals is_relevant = '__name__' in globs and \ globs["__name__"].startswith("unittest") del globs return is_relevant }}} that's what inheritance is for ;o) ... but quite probably that's not gonna happen, just a comment . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Ubuntu sustituye GIMP por F-Spot - http://feedproxy.google.com/~r/simelo-es/~3/-g48D6T6Ojs/ubuntu-sustituye-gimp-por-f-spot.html From martin at v.loewis.de Thu Jan 7 21:24:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:24:32 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B45543C.2090200@gmail.com> <1afaf6161001061932p9e7470esc6cdf7953b8c02fd@mail.gmail.com> Message-ID: <4B464300.2020204@v.loewis.de> >> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question about releasing the > GIL during regexp searches? No, because the regex engine may also operate on buffers that start moving around when you release the GIL. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:27:09 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:27:09 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B46439D.9030604@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in place - if their contents changes, regex results may be confusing. Regards, Martin From martin at v.loewis.de Thu Jan 7 21:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:31:21 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> Message-ID: <4B464499.4020305@v.loewis.de> > I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process a file > called ast.txt. Where does this file come from ? Is it generated from > Python.asdl ? astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. HTH, Martin From foom at fuhm.net Thu Jan 7 21:36:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 7 Jan 2010 15:36:37 -0500 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B46439D.9030604@v.loewis.de> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> On Jan 7, 2010, at 3:27 PM, Martin v. L?wis wrote: >> I've been wondering whether it's possible to release the GIL in the >> regex engine during matching. > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Even if they stay in place - > if their contents changes, regex results may be confusing. It seems probably worthwhile to optimize for the common case of using the regexp engine on an immutable object of type "str" or "bytes", and allow releasing the GIL in *that* case, even if you have to keep it for the general case. James From martin at v.loewis.de Thu Jan 7 21:45:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:45:42 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <82C03488-5D03-4E67-8B67-CE6EE5879D2B@fuhm.net> Message-ID: <4B4647F6.2060309@v.loewis.de> >>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even if they stay in place - >> if their contents changes, regex results may be confusing. > > It seems probably worthwhile to optimize for the common case of using > the regexp engine on an immutable object of type "str" or "bytes", and > allow releasing the GIL in *that* case, even if you have to keep it for > the general case. Right. This problem was the one that I thought of first. Thinking about these things is fairly difficult (to me, at least), so I think I could only tell whether I would consider a patch thread-safe that released the GIL around matching under selected circumstances - if I had the patch available. I don't see any obvious reason (assuming Guido's list of conditions holds - i.e. you are holding references to everything you access). Regards, Martin From ncoghlan at gmail.com Thu Jan 7 21:48:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Jan 2010 06:48:05 +1000 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45DECB.7070907@agama.tv> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> <4B45DECB.7070907@agama.tv> Message-ID: <4B464885.40806@gmail.com> Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a contributor agreement was essential for this particular contribution would depend on how much new code was needed for the backport, but the bulk of the copyright on the C RLock code would remain with Antoine regardless. However, sorting through the legalities of the contributor agreement really is the best way to make sure every is squared away nice and neatly from a legal point of view. After all, even if I was a lawyer (which I'm not, I'm just a developer with an interest in licensing issues), I still wouldn't be *your* lawyer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Jan 7 21:43:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Jan 2010 20:43:17 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?GIL_required_for_=5Fall=5F_Python_calls=3F?= References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the representation isn't moved away until the buffer is released. Regards Antoine. From martin at v.loewis.de Thu Jan 7 21:59:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 21:59:29 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: <4B45500C.8090905@mrabarnett.plus.com> References: <4B45500C.8090905@mrabarnett.plus.com> Message-ID: <4B464B31.7040406@v.loewis.de> > I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin From johan.gill at agama.tv Thu Jan 7 14:16:59 2010 From: johan.gill at agama.tv (Johan Gill) Date: Thu, 07 Jan 2010 14:16:59 +0100 Subject: [Python-Dev] Backported faster RLock to Python 2.6. In-Reply-To: <4B45D22F.40404@gmail.com> References: <4B45AD8C.5000405@agama.tv> <319e029f1001070312q136bd933u669c1291fcb1b602@mail.gmail.com> <4B45D22F.40404@gmail.com> Message-ID: <4B45DECB.7070907@agama.tv> On 01/07/2010 01:23 PM, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open > source license without requesting additional permission from the > copyright holder. For corporate contributions, I believe the contributor > agreement needs to be signed by an authorised agent of the company - the > place to check that would probably be psf at python.org (that's the email > address for the PSF board). > > Assuming the subject line relates to the code that you would like to > contribute though, that particular change is unlikely to happen - 2.6 is > in maintenance mode and changing RLock from a Python implementation to > the faster C one is solidly in new feature territory. Although a > backport of the 3.2 C RLock implementation to 2.7 could be useful, I > doubt that backporting code provided by an existing committer would be > the subject of this query :) > > Regards, > Nick. > > Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Regards Johan Gill Agama Technologies From yoann.padioleau at facebook.com Thu Jan 7 22:07:55 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Thu, 7 Jan 2010 13:07:55 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464499.4020305@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> Message-ID: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> On Jan 7, 2010, at 12:31 PM, Martin v. L?wis wrote: >> I would like to use astgen.py to generate python classes corresponding to the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take as an argument a .asdl file, but apparently it instead process a file >> called ast.txt. Where does this file come from ? Is it generated from >> Python.asdl ? > > astgen.py is not used to process asdl files; ast.txt lives right next to > astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a reflection mechanism things that were generated from ast.txt, then there could be a mismatch. Where does ast.txt comes from ? Shouldn't it be generated itself from Python.adsl ? So we would have Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py containing all the UnarySub, Expression, classes that represents a Python AST. > > HTH, > Martin From martin at v.loewis.de Thu Jan 7 22:11:36 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 07 Jan 2010 22:11:36 +0100 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> Message-ID: <4B464E08.5020703@v.loewis.de> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the representation isn't moved away until the buffer is > released. In 2.7, we currently get the buffer with bf_getreadbuffer. In 3.x, we have /* Release the buffer immediately --- possibly dangerous but doing something else would require some re-factoring */ PyBuffer_Release(&view); Even if we do use the new API, and correctly, it still might be confusing if the contents of the buffer changes underneath. Regards, Martin From martin at v.loewis.de Thu Jan 7 22:16:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Jan 2010 22:16:12 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> Message-ID: <4B464F1C.7020404@v.loewis.de> >> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose as a reflection mechanism things that were generated from > ast.txt, then there could be a mismatch. Where does ast.txt comes > from ? Shouldn't it be generated itself from Python.adsl ? What you may not be aware of is that Tools/compiler (and the compiler package that it builds on) are both unused and unmaintained. If the package stops working correctly - tough luck. > So we would have > > Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py > containing all the UnarySub, Expression, classes that represents a > Python AST. No - what actually happens in Python 3.x is this: both the compiler package and Tools/compiler are removed. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 01:10:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 01:10:35 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM Message-ID: <201001080110.35800.victor.stinner@haypocalc.com> Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading an UTF-8 text file including a BOM: #7185 (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with the UTF-8-SIG encoding, but it's possible to do better. I propose to improve open() (TextIOWrapper) by using the BOM to choose the right encoding. I think that only files opened in read only mode should support this new feature. *Read* the BOM in a *write* only file would cause unexpected behaviours. Since my proposition changes the result TextIOWrapper.read()/readline() for files starting with a BOM, we might introduce an option to open() to enable the new behaviour. But is it really needed to keep the backward compatibility? I wrote a proof of concept attached to the issue #7651. My patch only changes the behaviour of TextIOWrapper for reading files starting with a BOM. It doesn't work yet if a seek() is used before the first read. -- Victor Stinner http://www.haypocalc.com/ From guido at python.org Fri Jan 8 01:52:20 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 16:52:20 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner wrote: > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". > > See recent issues related to reading an UTF-8 text file including a BOM: #7185 > (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with > the UTF-8-SIG encoding, but it's possible to do better. > > I propose to improve open() (TextIOWrapper) by using the BOM to choose the > right encoding. I think that only files opened in read only mode should > support this new feature. *Read* the BOM in a *write* only file would cause > unexpected behaviours. > > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? > > I wrote a proof of concept attached to the issue #7651. My patch only changes > the behaviour of TextIOWrapper for reading files starting with a BOM. It > doesn't work yet if a seek() is used before the first read. > > -- > Victor Stinner > http://www.haypocalc.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Fri Jan 8 03:23:08 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 02:23:08 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46970C.9010306@mrabarnett.plus.com> Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > Alternatively, have a universal UTF-8/16/32 encoding, ie one that expects UTF-8, with or without BOM, or UTF-16/32 with BOM. > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> >> See recent issues related to reading an UTF-8 text file including a BOM: #7185 >> (csv) and #7519 (ConfigParser). Such file can be opened in unicode mode with >> the UTF-8-SIG encoding, but it's possible to do better. >> >> I propose to improve open() (TextIOWrapper) by using the BOM to choose the >> right encoding. I think that only files opened in read only mode should >> support this new feature. *Read* the BOM in a *write* only file would cause >> unexpected behaviours. >> >> Since my proposition changes the result TextIOWrapper.read()/readline() for >> files starting with a BOM, we might introduce an option to open() to enable >> the new behaviour. But is it really needed to keep the backward compatibility? >> >> I wrote a proof of concept attached to the issue #7651. My patch only changes >> the behaviour of TextIOWrapper for reading files starting with a BOM. It >> doesn't work yet if a seek() is used before the first read. >> From nick.bastin at gmail.com Fri Jan 8 04:11:03 2010 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 7 Jan 2010 22:11:03 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> Message-ID: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs to do this on FreeBSD as well (adding bsd to the list of platforms for which this is performed "solves" the problem, but I don't pretend to know enough about either distutils or freebsd to determine if this is the correct solution). -- Nick From glyph at twistedmatrix.com Fri Jan 8 04:34:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 7 Jan 2010 22:34:36 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? It *is* crazy, but unfortunately rather common. Wikipedia has a good description of the issues: . Basically, some Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as being UTF-8, so it's become a convention to do that. That's not good enough, so you need to guess the encoding as well to make sure, but if there is a BOM and you can otherwise verify that the file is probably UTF-8 encoded, you should discard it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Jan 8 05:17:28 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 07 Jan 2010 23:17:28 -0500 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solaris and automatically adds '.' to the library_dirs, and > I suspect it just needs to do this on FreeBSD as well (adding bsd to > the list of platforms for which this is performed "solves" the > problem, but I don't pretend to know enough about either distutils or > freebsd to determine if this is the correct solution). I wouldn't say it needed discussion on the SIG: just create a bug report, with the tentative patch you have worked out, and get it assigned to Tarek. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGsdQACgkQ+gerLs4ltQ5BMQCgtV8snMXH/6dDwgdN4sIJljLd koYAoKq6c0tKsRSrITHcygu4Od9FVzF5 =BJaE -----END PGP SIGNATURE----- From guido at python.org Fri Jan 8 05:21:04 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 7 Jan 2010 20:21:04 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > > file starting with a BOM, read()/readline() returns also the BOM whereas the > > BOM should be "ignored". > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > > It *is* crazy, but unfortunately rather common. ?Wikipedia has a good > description of the issues: > . ?Basically, some > Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as > being UTF-8, so it's become a convention to do that. ?That's not good > enough, so you need to guess the encoding as well to make sure, but if there > is a BOM and you can otherwise verify that the file is probably UTF-8 > encoded, you should discard it. That doesn't make sense. If the file isn't UTF-8 you can't see the BOM, because the BOM itself is UTF-8-encoded. (And yes, I know this happens. Doesn't mean we need to auto-guess by default; there are lots of issues e.g. what should happen after seeking to offset 0?) -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Fri Jan 8 07:06:16 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 08 Jan 2010 15:06:16 +0900 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from the user for each instance. But it should do something sane when reading such files. I can't really see any harm in throwing it away, especially since use of ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated IIRC. From tseaver at palladion.com Fri Jan 8 07:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 01:12:12 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> >> BOM should be "ignored". >> >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. > > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) The BOM should not be seekeable if the file is opened with the proposed "guess encoding from BOM" mode: it isn't properly part of the stream at all in that case. A UTF-8 BOM is an absurditiy, but it exists *everywhere* in the wild: Python would do wll to make it as easy as possible to consume such files, as well as the non-insane versions (UTF-16 / UTF-32 BOMs). In the best of all possible worlds, I would just try opening the file so: f = open('/path/to/file', 'r', encoding="DWIFM") and any BOM present would set the encoding for the remainder of the stream.. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGzLsACgkQ+gerLs4ltQ5+cwCdGfycPdj6+cPfD23vH644SpHL sI0AoLGD7nfgMEJdJhBr90yjQQHfDgcJ =js+2 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Fri Jan 8 08:55:27 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 8 Jan 2010 02:55:27 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. It's just that the UTF-8 decoding of the BOM at the start of a file should be "". > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I think it's pretty clear that the BOM should still be skipped in that case ... From martin at v.loewis.de Fri Jan 8 10:05:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:05:17 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F54D.9090103@v.loewis.de> >> It *is* crazy, but unfortunately rather common. Wikipedia has a good >> description of the issues: >> . Basically, some >> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >> being UTF-8, so it's become a convention to do that. That's not good >> enough, so you need to guess the encoding as well to make sure, but if there >> is a BOM and you can otherwise verify that the file is probably UTF-8 >> encoded, you should discard it. > > That doesn't make sense. If the file isn't UTF-8 you can't see the > BOM, because the BOM itself is UTF-8-encoded. I think what Glyph meant is this: if a file starts with the UTF-8 signature, assume it's UTF-8. Then validate the assumption against the rest of the file also, and then process it as UTF-8. If the rest clearly is not UTF-8, assume that the UTF-8 signature is bogus. I understood this proposal as a general processing guideline, not something the io library should do (but, say, a text editor). FWIW, I'm personally in favor of using the UTF-8 signature. If people consider them crazy talk, that may be because UTF-8 can't possibly have a byte order - hence I call it a signature, not the BOM. As a signature, I don't consider it crazy at all. There is a long tradition of having magic bytes in files (executable files, Postscript, PDF, ... - see /etc/magic). Having a magic byte sequence for plain text to denote the encoding is useful and helps reducing moji-bake. This is the reason it's used on Windows: notepad would normally assume that text is in the ANSI code page, and for compatibility, it can't stop doing that. So the UTF-8 signature gives them an exit strategy. Regards, Martin From martin at v.loewis.de Fri Jan 8 10:06:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:06:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <201001080110.35800.victor.stinner@haypocalc.com> <87fx6hjfl3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4B46F59A.5060905@v.loewis.de> > But it should do something sane when reading such files. I can't > really see any harm in throwing it away, especially since use of > ZERO-WIDTH NO-BREAK SPACE as a joining character has been deprecated > IIRC. And indeed it does, when you open the file in the utf-8-sig encoding. Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 10:08:30 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 10:08:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46970C.9010306@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46970C.9010306@mrabarnett.plus.com> Message-ID: <201001081008.30904.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 03:23:08, MRAB a ?crit : > Guido van Rossum wrote: > > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > > talk. And for the other two, perhaps it would make more sense to have > > a separate encoding-guessing function that takes a binary stream and > > returns a text stream wrapping it with the proper encoding? > > Alternatively, have a universal UTF-8/16/32 encoding, ie one that > expects UTF-8, > with or without BOM, or UTF-16/32 with BOM. Do you mean open(filename, encoding="BOM")? I suppose that "BOM" would be a magical value specific to read a text file (open(filename, "r")), not a real codec? Otherwise which encoding should be used for open(filename, "w", encoding="BOM")? -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Fri Jan 8 10:10:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:10:23 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001080110.35800.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <4B46F67F.60604@v.loewis.de> > Builtin open() function is unable to open an UTF-16/32 file starting with a > BOM if the encoding is not specified (raise an unicode error). For an UTF-8 > file starting with a BOM, read()/readline() returns also the BOM whereas the > BOM should be "ignored". It depends. If you use the utf-8-sig encoding, it *will* ignore the UTF-8 signature. > Since my proposition changes the result TextIOWrapper.read()/readline() for > files starting with a BOM, we might introduce an option to open() to enable > the new behaviour. But is it really needed to keep the backward compatibility? Absolutely. And there is no need to produce a new option, but instead use the existing options: define an encoding that auto-detects the encoding from the family of BOMs. Maybe you call it encoding="sniff". Regards, Martin From martin at v.loewis.de Fri Jan 8 10:11:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Jan 2010 10:11:51 +0100 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> Message-ID: <4B46F6D7.1050301@v.loewis.de> Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. I'm fairly skeptical that anybody on distutils SIG is interested in details of the Python build process... Regards, Martin From victor.stinner at haypocalc.com Fri Jan 8 11:27:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:27:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081127.44044.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : (...) > (And yes, I know this happens. Doesn't mean we need to auto-guess by > default; there are lots of issues e.g. what should happen after > seeking to offset 0?) I wrote a new version of my patch (version 3): * don't change the default behaviour: use open(filename, encoding="BOM") to check the BOM is there is any * fix for seek(0): always ignore the BOM * add an unit test: check that the right encoding is detect, but also the the BOM is ignored (especially after a seek(0)) BOM encoding doesn't work for writing into a file, so open(filename, "w", encoding="BOM") raises a ValueError. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:31:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:31:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <201001081131.37492.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 01:52:20, Guido van Rossum a ?crit : > And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? I choosed to modify open()+TextIOWrapper instead of writing a new function because I would like to avoid an extra read operation (syscall) on the file. With my implementation, no specific read operation is needed to detect the BOM. The BOM is simply checked in the first _read_chunk(). -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Jan 8 11:40:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 11:40:28 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Improve_open=28=29_to_support_reading?= =?iso-8859-1?q?_file_starting_with=09an_unicode_BOM?= In-Reply-To: <4B46F67F.60604@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> Message-ID: <201001081140.28124.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : > > Builtin open() function is unable to open an UTF-16/32 file starting with > > a BOM if the encoding is not specified (raise an unicode error). For an > > UTF-8 file starting with a BOM, read()/readline() returns also the BOM > > whereas the BOM should be "ignored". > > It depends. If you use the utf-8-sig encoding, it *will* ignore the > UTF-8 signature. Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to remove the BOM after the first read (much harder if you use a module like ConfigParser or csv). > > Since my proposition changes the result TextIOWrapper.read()/readline() > > for files starting with a BOM, we might introduce an option to open() to > > enable the new behaviour. But is it really needed to keep the backward > > compatibility? > > Absolutely. And there is no need to produce a new option, but instead > use the existing options: define an encoding that auto-detects the > encoding from the family of BOMs. Maybe you call it encoding="sniff". Good idea, I choosed open(filename, encoding="BOM"). -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri Jan 8 15:27:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:27:42 +0000 (UTC) Subject: [Python-Dev] GIL required for _all_ Python calls? References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: > > Even if we do use the new API, and correctly, it still might be > confusing if the contents of the buffer changes underneath. Well, no more confusing than when you compute a SHA1 hash or zlib- compress the buffer, is it? Regards Antoine From solipsis at pitrou.net Fri Jan 8 15:34:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 14:34:26 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any Well, I think if we implement this the default behaviour *should* be changed. It looks a bit senseless to have two different "auto-choose" options, one with encoding=None and one with encoding="BOM". Regards Antoine. From guido at python.org Fri Jan 8 16:48:44 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:48:44 -0800 Subject: [Python-Dev] GIL required for _all_ Python calls? In-Reply-To: References: <4B45500C.8090905@mrabarnett.plus.com> <4B46439D.9030604@v.loewis.de> <4B464E08.5020703@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 6:27 AM, Antoine Pitrou wrote: > Le Thu, 07 Jan 2010 22:11:36 +0100, Martin v. L?wis a ?crit?: >> >> Even if we do use the new API, and correctly, it still might be >> confusing if the contents of the buffer changes underneath. > > Well, no more confusing than when you compute a SHA1 hash or zlib- > compress the buffer, is it? That depends. Algorithms that make exactly one pass over the buffer will run fine (maybe producing a meaningless result). But the regex matcher may scan the buffer repeatedly (for backtracking purposes) and it would take a considerable analysis to prove that cannot mess up its internal data structures if the data underneath changes. (I give it a decent chance that it's fine, but since it was written without ever considering this possibility I'm not 100% sure.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:52:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:52:48 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 11:55 PM, Glyph Lefkowitz wrote: > I'm saying that the BOM itself isn't enough to detect that the file is actually UTF-8. And I'm saying that it is, with as much certainty as we can ever guess the encoding of a file. > If (for whatever reason: explicitly specified, guessed in some other way) the file's encoding is determined to be something else, the bytes comprising the BOM should be decoded as normal. ?It's just that the UTF-8 decoding of the BOM at the start of a file should be "". Sure, a Latin-1-encoded file could start with the same pattern that is a UTF-8-encoded BOM. But at that point, a UTF-16-encoded file is also valid Latin-1. The question was in the context of encoding-guessing; if we're guessing, a UTF-8-encoded BOM cannot signify anything else but UTF-8. (Ditto for UTF-16 and UTF-32 BOMs.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:54:15 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:54:15 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> >> I wrote a new version of my patch (version 3): >> >> ?* don't change the default behaviour: use open(filename, encoding="BOM") to >> check the BOM is there is any > > Well, I think if we implement this the default behaviour *should* be changed. > It looks a bit senseless to have two different "auto-choose" options, one with > encoding=None and one with encoding="BOM". Well there *are* two different auto options: use the environment variables (LANG etc.) or inspect the contents of the file. I think it would be useful to have ways to specify both. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:56:46 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:56:46 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Fri, Jan 8, 2010 at 1:05 AM, "Martin v. L?wis" wrote: >>> It *is* crazy, but unfortunately rather common. ?Wikipedia has a good >>> description of the issues: >>> . ?Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. ?That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. > > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Sure. I said "crazy talk" only to stir up discussion. Which worked. :-) Also, I don't want Python's default behavior to change -- sniffing the encoding should be a separate option. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Jan 8 16:59:45 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Jan 2010 07:59:45 -0800 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: > The BOM should not be seekeable if the file is opened with the proposed > "guess encoding from BOM" mode: ?it isn't properly part of the stream at > all in that case. This feels about right to me. There are still questions though: immediately after opening a file with a BOM, what should .tell() return? And regardless of that, .seek(0) should put the file in that same initial state. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Jan 8 17:03:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:03:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > > Well, I think if we implement this the default behaviour *should* be changed. > > It looks a bit senseless to have two different "auto-choose" options, one with > > encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Yes, perhaps. In the context of open() however I think it would be helpful to change the default. Moreover, reading the BOM is certainly much more reliable than our current guessing based on the locale or the "device encoding". Regards Antoine. From mal at egenix.com Fri Jan 8 17:25:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 17:25:22 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B475C72.1010207@egenix.com> Guido van Rossum wrote: > On Fri, Jan 8, 2010 at 6:34 AM, Antoine Pitrou wrote: >> Victor Stinner haypocalc.com> writes: >>> >>> I wrote a new version of my patch (version 3): >>> >>> * don't change the default behaviour: use open(filename, encoding="BOM") to >>> check the BOM is there is any >> >> Well, I think if we implement this the default behaviour *should* be changed. >> It looks a bit senseless to have two different "auto-choose" options, one with >> encoding=None and one with encoding="BOM". > > Well there *are* two different auto options: use the environment > variables (LANG etc.) or inspect the contents of the file. I think it > would be useful to have ways to specify both. Shouldn't this encoding guessing be a separate function that you call on either a file or a seekable stream ? After all, detecting encodings is just as useful to have for non-file streams. You'd then avoid having to stuff everything into a single function call and also open up the door for more complex application specific guess work or defaults. The whole process would then have two steps: 1. guess encoding import codecs encoding = codecs.guess_file_encoding(filename) 2. open the file with the found encoding f = open(filename, encoding=encoding) For seekable streams f, you'd have: 1. guess encoding import codecs encoding = codecs.guess_stream_encoding(f) 2. wrap the stream with a reader for the found encoding reader_class = codecs.getreader(encoding) g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Jan 8 17:31:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Jan 2010 16:31:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Improve_open=28=29_to_support_reading_file?= =?utf-8?q?_starting=09with_an_unicode_BOM?= References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver palladion.com> wrote: > > The BOM should not be seekeable if the file is opened with the proposed > > "guess encoding from BOM" mode: it isn't properly part of the stream at > > all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? tell() in the context of text I/O is specified to return an "opaque cookie". So whatever value it returns would probably be fine, as long as seeking to that value leaves the file in an acceptable state. Rewinding (seeking to 0) in the presence of a BOM is already reasonably supported by the TextIOWrapper object: >>> dec = codecs.getincrementaldecoder('utf-16')() >>> dec.decode(b'\xff\xfea\x00b\x00') 'ab' >>> dec.decode(b'\xff\xfea\x00b\x00') '\ufeffab' >>> >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' There are tests for this in test_io.py (test_encoded_writes, line 1929, and test_append_bom and test_seek_bom, line 2045). Regards Antoine. From python at mrabarnett.plus.com Fri Jan 8 17:47:18 2010 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 08 Jan 2010 16:47:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081127.44044.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> Message-ID: <4B476196.4080409@mrabarnett.plus.com> Victor Stinner wrote: > Le vendredi 08 janvier 2010 05:21:04, Guido van Rossum a ?crit : > (...) >> (And yes, I know this happens. Doesn't mean we need to auto-guess by >> default; there are lots of issues e.g. what should happen after >> seeking to offset 0?) > > I wrote a new version of my patch (version 3): > > * don't change the default behaviour: use open(filename, encoding="BOM") to > check the BOM is there is any > * fix for seek(0): always ignore the BOM > * add an unit test: check that the right encoding is detect, but also the the > BOM is ignored (especially after a seek(0)) > > BOM encoding doesn't work for writing into a file, so open(filename, "w", > encoding="BOM") raises a ValueError. > I think it's similar to universal newline mode. You can tell it that you're reading UTF-something-encoded text (common forms only). The preference is UTF-8, but it could be UTF-8-sig (from Windows), or possibly UTF-16/32, which really need a BOM because there are multiple bytes per codepoint, so the bytes could be big-endian or little-endian. The BOM (or signature) tells you what the encoding is, defaulting to UTF-8 if there's none. If it subsequently raises a DecodeError, then so be it! Maybe there should also be a way of determining what encoding it decided it was, so that you can then write a new file in that same encoding. From status at bugs.python.org Fri Jan 8 18:07:13 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Jan 2010 18:07:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100108170714.014B078669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/01/10 - 01/08/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2544 open (+27) / 16937 closed (+15) / 19481 total (+42) Open issues with patches: 1017 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2509 (+27) pending 34 ( +0) Issues Created Or Reopened (43) _______________________________ Extended slicing with classic class behaves strangely 01/07/10 http://bugs.python.org/issue7532 reopened mark.dickinson patch optparse library documentation has an insignificant formatting i 01/01/10 CLOSED http://bugs.python.org/issue7618 created vazovsky patch imaplib shouldn't use cause DeprecationWarnings in 2.6 01/01/10 CLOSED http://bugs.python.org/issue7619 created djc Vim syntax highlight 01/02/10 http://bugs.python.org/issue7620 created july patch Test issue 01/02/10 CLOSED http://bugs.python.org/issue7621 created georg.brandl [patch] improve unicode methods: split() rsplit() and replace() 01/03/10 http://bugs.python.org/issue7622 created flox patch PropertyType missing in Lib/types.py 01/03/10 CLOSED http://bugs.python.org/issue7623 created wplappert isinstance(... ,collections.Callable) fails with oldstyle class 01/03/10 http://bugs.python.org/issue7624 created rgammans bytearray needs more tests for "b.some_method()[0] is not b" 01/03/10 http://bugs.python.org/issue7625 created flox patch Entity references without semicolon in HTMLParser 01/03/10 CLOSED http://bugs.python.org/issue7626 created stefan.schweizer mailbox.MH.remove() lock handling is broken 01/04/10 http://bugs.python.org/issue7627 created sraustein round() doesn't work correctly in 3.1.1 01/04/10 CLOSED http://bugs.python.org/issue7628 created bkovt Compiling with mingw32 gcc, content of variable "extra_postargs" 01/04/10 CLOSED http://bugs.python.org/issue7629 created popelkopp Strange behaviour of decimal.Decimal 01/04/10 CLOSED http://bugs.python.org/issue7630 created parmax undefined label: bltin-file-objects 01/04/10 CLOSED http://bugs.python.org/issue7631 created ezio.melotti dtoa.c: oversize b in quorem 01/04/10 http://bugs.python.org/issue7632 created skrah decimal.py: type conversion in context methods 01/04/10 http://bugs.python.org/issue7633 created skrah patch, easy next/previous links in documentation skip some sections 01/05/10 CLOSED http://bugs.python.org/issue7634 created gagenellina 19.6 xml.dom.pulldom doc: stub? 01/05/10 http://bugs.python.org/issue7635 created tjreedy Add a set update action to optparse 01/05/10 CLOSED http://bugs.python.org/issue7636 created hardkrash patch Improve 19.5. xml.dom.minidom doc 01/05/10 http://bugs.python.org/issue7637 created tjreedy Counterintuitive str.splitlines() inconsistency. 01/05/10 CLOSED http://bugs.python.org/issue7638 created vencabot_teppoo bdist_msi fails on files with long names 01/05/10 http://bugs.python.org/issue7639 created mmm77 buffered io seek() buggy 01/05/10 http://bugs.python.org/issue7640 created pakal patch Built-in Formatter accepts undocumented presentation type 01/06/10 CLOSED http://bugs.python.org/issue7641 created lrekucki [patch] Minor improvement in os.system doc 01/06/10 http://bugs.python.org/issue7642 created Val patch What is an ASCII linebreak? 01/06/10 http://bugs.python.org/issue7643 created flox bug in nntplib.body() method with possible fix 01/06/10 http://bugs.python.org/issue7644 created mdmullins easy test_distutils fails on Windows XP 01/06/10 http://bugs.python.org/issue7645 created austin987 test_telnetlib fails in Windows XP 01/06/10 http://bugs.python.org/issue7646 created austin987 Add statvfs flags to the posix module 01/06/10 http://bugs.python.org/issue7647 created ajax at redhat.com patch test_urllib2 fails on Windows if not run from C: 01/06/10 http://bugs.python.org/issue7648 created austin987 easy "u'%c' % char" broken for chars in range '\x80'-'\xFF' 01/07/10 http://bugs.python.org/issue7649 created ezio.melotti patch, easy test_uuid is invalid 01/07/10 http://bugs.python.org/issue7650 created austin987 Python3: guess text file charset using the BOM 01/07/10 http://bugs.python.org/issue7651 created haypo patch Merge C version of decimal into py3k. 01/07/10 http://bugs.python.org/issue7652 created mark.dickinson Fix description of the way the PythonPath Windows registry key w 01/07/10 CLOSED http://bugs.python.org/issue7653 created BoarGules patch, needs review [patch] Enable additional bytes and memoryview tests. 01/07/10 http://bugs.python.org/issue7654 created flox patch PEP 374 Title usage & script redirection typo fixes 01/07/10 CLOSED http://bugs.python.org/issue7655 created bab9e9 patch test_hashlib fails on some installations (specifically Neal's re 01/08/10 http://bugs.python.org/issue7656 created r.david.murray test_ctypes failure on AIX 5.3 01/08/10 http://bugs.python.org/issue7657 created mallayya patch OS X pythonw.c compile error with 10.4 or earlier deployment tar 01/08/10 http://bugs.python.org/issue7658 created ned.deily Problems with attribute assignment on object instances 01/08/10 http://bugs.python.org/issue7659 created pakal Issues Now Closed (40) ______________________ shutil fails when copying to NTFS in Linux 762 days http://bugs.python.org/issue1545 benjamin.peterson patch Test 751 days http://bugs.python.org/issue1619 georg.brandl Windows Registry issue with 2.5.2 AMD64 msi 645 days http://bugs.python.org/issue2539 loewis Extension module build fails for MinGW: missing vcvarsall.bat 618 days http://bugs.python.org/issue2698 Daniel26 optparse print_usage(),.. methods are not documented 542 days http://bugs.python.org/issue3340 ezio.melotti reference leaks in test_distutils 500 days http://bugs.python.org/issue3660 ezio.melotti patch _sha256 et al. encode to UTF-8 by default 20 days http://bugs.python.org/issue3745 lemburg 26backport Add Option to Bind to a Local IP Address in httplib.py 464 days http://bugs.python.org/issue3972 gregory.p.smith patch, easy no reference for optparse methods 388 days http://bugs.python.org/issue4635 ezio.melotti PyArg_Parse* should raise TypeError for float parsed with intege 339 days http://bugs.python.org/issue5080 mark.dickinson patch Don't use PyLong_SHIFT with _PyLong_AsScaledDouble() 282 days http://bugs.python.org/issue5576 mark.dickinson patch PyFrame_GetLineNumber 244 days http://bugs.python.org/issue5954 georg.brandl patch PyCode_NewEmpty 244 days http://bugs.python.org/issue5959 georg.brandl patch Add non-command help topics to help completion of cmd.Cmd 241 days http://bugs.python.org/issue5991 georg.brandl patch make error 231 days http://bugs.python.org/issue6067 georg.brandl no longer possible to hash arrays 229 days http://bugs.python.org/issue6071 benjamin.peterson patch zipfile: Invalid argument when opening zero-sized files 173 days http://bugs.python.org/issue6511 r.david.murray use different mechanism for pythonw on osx 127 days http://bugs.python.org/issue6834 ned.deily easy Py3k doc: "from __future__ import division" not necessary 32 days http://bugs.python.org/issue7432 ezio.melotti cPickle: stack underflow in load_pop() 31 days http://bugs.python.org/issue7455 pitrou patch crash in str.rfind() with an invalid start value 25 days http://bugs.python.org/issue7458 pitrou patch Implement fastsearch algorithm for rfind/rindex 26 days http://bugs.python.org/issue7462 ezio.melotti patch GZipFile.readline too slow 20 days http://bugs.python.org/issue7471 pitrou patch ssl module documentation: SSLSocket.unwrap description shown twi 5 days http://bugs.python.org/issue7592 georg.brandl Installation - 2 tests failed: test_cmd_line test_xmlrpc 7 days http://bugs.python.org/issue7601 ezio.melotti optparse library documentation has an insignificant formatting i 2 days http://bugs.python.org/issue7618 ezio.melotti patch imaplib shouldn't use cause DeprecationWarnings in 2.6 1 days http://bugs.python.org/issue7619 djc Test issue 0 days http://bugs.python.org/issue7621 ezio.melotti PropertyType missing in Lib/types.py 0 days http://bugs.python.org/issue7623 benjamin.peterson Entity references without semicolon in HTMLParser 2 days http://bugs.python.org/issue7626 r.david.murray round() doesn't work correctly in 3.1.1 0 days http://bugs.python.org/issue7628 benjamin.peterson Compiling with mingw32 gcc, content of variable "extra_postargs" 0 days http://bugs.python.org/issue7629 r.david.murray Strange behaviour of decimal.Decimal 0 days http://bugs.python.org/issue7630 mark.dickinson undefined label: bltin-file-objects 0 days http://bugs.python.org/issue7631 pitrou next/previous links in documentation skip some sections 0 days http://bugs.python.org/issue7634 georg.brandl Add a set update action to optparse 3 days http://bugs.python.org/issue7636 r.david.murray patch Counterintuitive str.splitlines() inconsistency. 0 days http://bugs.python.org/issue7638 flox Built-in Formatter accepts undocumented presentation type 0 days http://bugs.python.org/issue7641 eric.smith Fix description of the way the PythonPath Windows registry key w 0 days http://bugs.python.org/issue7653 r.david.murray patch, needs review PEP 374 Title usage & script redirection typo fixes 0 days http://bugs.python.org/issue7655 georg.brandl patch Top Issues Most Discussed (10) ______________________________ 22 [patch] improve unicode methods: split() rsplit() and replace() 5 days open http://bugs.python.org/issue7622 14 Test suite emits many DeprecationWarnings when -3 is enabled 91 days open http://bugs.python.org/issue7092 13 unicode_escape codec does not escape quotes 8 days open http://bugs.python.org/issue7615 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 0 days open http://bugs.python.org/issue7658 8 Merge C version of decimal into py3k. 1 days open http://bugs.python.org/issue7652 8 "u'%c' % char" broken for chars in range '\x80'-'\xFF' 2 days open http://bugs.python.org/issue7649 8 decimal.py: type conversion in context methods 4 days open http://bugs.python.org/issue7633 8 Patch for [ 735515 ] urllib2 should cache 301 redir 906 days open http://bugs.python.org/issue1755841 7 Test issue 0 days closed http://bugs.python.org/issue7621 7 Cannot use both read and readline method in same ZipExtFile obj 8 days open http://bugs.python.org/issue7610 From tseaver at palladion.com Fri Jan 8 22:09:54 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:09:54 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 10:12 PM, Tres Seaver wrote: >> The BOM should not be seekeable if the file is opened with the proposed >> "guess encoding from BOM" mode: it isn't properly part of the stream at >> all in that case. > > This feels about right to me. There are still questions though: > immediately after opening a file with a BOM, what should .tell() > return? And regardless of that, .seek(0) should put the file in that > same initial state. I think the behavior should be something like: >>> f = open('/path/to/maybe-BOM-encoded-file', 'r', encoding='BOM') >>> f.tell() 0L >>> f.seek(-1) >>> f.tell() # count of unicode chars in decoded stream 45L >>> f.seek(0) >>> f.read(1) # read first unicode char decoded from stream. 'A' In other words, the BOM is not readable / seekable at all: it is invisible to the consumer of the decoded stream. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHnyIACgkQ+gerLs4ltQ6s3QCgznD+7FbUzfCbe5TS6OcoXjMg rdgAoJAMEXe2xwLCIwJaZ6XA6rVyTIAi =oXb3 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:19:10 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:19:10 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B475C72.1010207@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: > Shouldn't this encoding guessing be a separate function that you call > on either a file or a seekable stream ? > > After all, detecting encodings is just as useful to have for non-file > streams. Other stream sources typically have out-of-band ways to signal the encoding: only when reading from the filesystem do we pretty much *have* to guess, and in that case the BOM / signature is the best heuristic we have. Also, some non-file streams are not seekable, and so can't be guessed via a pre-pass. > You'd then avoid having to stuff everything into > a single function call and also open up the door for more complex > application specific guess work or defaults. > > The whole process would then have two steps: > > 1. guess encoding > > import codecs > encoding = codecs.guess_file_encoding(filename) Filename is not enough information: or do you mean that API to actually open the stream? > 2. open the file with the found encoding > > f = open(filename, encoding=encoding) > > For seekable streams f, you'd have: > > 1. guess encoding > > import codecs > encoding = codecs.guess_stream_encoding(f) > > 2. wrap the stream with a reader for the found encoding > > reader_class = codecs.getreader(encoding) > g = reader_class(f) > Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoU4ACgkQ+gerLs4ltQ5o3QCeLOJ7J91E+5f66vhgu1BUhYh4 9UgAnR2IeCd0BCsPez8ZilGNHJfhRn3Y =SoPb -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Jan 8 22:14:59 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:14:59 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B46F54D.9090103@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> It *is* crazy, but unfortunately rather common. Wikipedia has a good >>> description of the issues: >>> . Basically, some >>> Windows text APIs will emit a UTF-8 "BOM" in order to identify the file as >>> being UTF-8, so it's become a convention to do that. That's not good >>> enough, so you need to guess the encoding as well to make sure, but if there >>> is a BOM and you can otherwise verify that the file is probably UTF-8 >>> encoded, you should discard it. >> That doesn't make sense. If the file isn't UTF-8 you can't see the >> BOM, because the BOM itself is UTF-8-encoded. > > I think what Glyph meant is this: if a file starts with the UTF-8 > signature, assume it's UTF-8. Then validate the assumption against the > rest of the file also, and then process it as UTF-8. If the rest clearly > is not UTF-8, assume that the UTF-8 signature is bogus. If the programmer opens the file using a "guess using the BOM" encoding, Python should *not* attempt to verify that the file is properly encoded: it should check for (and consume) any BOM, and then return a stream which uses the encoding inferred from the BOM. Any errors should be handled later, when characters are read, exactly as if the file had been opened with the same encoding guessed from the BOM. > I understood this proposal as a general processing guideline, not > something the io library should do (but, say, a text editor). > > FWIW, I'm personally in favor of using the UTF-8 signature. If people > consider them crazy talk, that may be because UTF-8 can't possibly have > a byte order - hence I call it a signature, not the BOM. As a signature, > I don't consider it crazy at all. There is a long tradition of having > magic bytes in files (executable files, Postscript, PDF, ... - see > /etc/magic). Having a magic byte sequence for plain text to denote the > encoding is useful and helps reducing moji-bake. This is the reason it's > used on Windows: notepad would normally assume that text is in the ANSI > code page, and for compatibility, it can't stop doing that. So the UTF-8 > signature gives them an exit strategy. Agreed. Having that marker at the start of the file makes interop with other tools *much* easier. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHoFMACgkQ+gerLs4ltQ73dACffwUfyh6Q9vUnKYf367QFjNcU RRMAoNuKCWEx7j+MSdTv+UjhAPynBc14 =uAX6 -----END PGP SIGNATURE----- From eric at trueblade.com Fri Jan 8 22:40:47 2010 From: eric at trueblade.com (Eric Smith) Date: Fri, 8 Jan 2010 16:40:47 -0500 (EST) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? >> >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. But what if the file were in (for example) a zip file? I think you definitely want to have access to this functionality outside of open(). Eric. From foom at fuhm.net Fri Jan 8 22:49:23 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 8 Jan 2010 16:49:23 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: On Jan 8, 2010, at 4:14 PM, Tres Seaver wrote: >> I understood this proposal as a general processing guideline, not >> something the io library should do (but, say, a text editor). >> >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly >> have >> a byte order - hence I call it a signature, not the BOM. As a >> signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote >> the >> encoding is useful and helps reducing moji-bake. This is the reason >> it's >> used on Windows: notepad would normally assume that text is in the >> ANSI >> code page, and for compatibility, it can't stop doing that. So the >> UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop > with > other tools *much* easier. Putting the BOM at the beginning of UTF-8 text files is not a good idea, it makes interop much *worse* on a unix system, not better. Without the BOM, most commands do the right thing with UTF-8 text. E.g. to concatenate two files: $ cat file-1 file-2 > file-3 With a BOM at the beginning of the file, it won't work right. Of course, you could modify "cat" (and every other stream processing command) to know how to consume and emit BOMs, and omit the extra one that would show up in the middle of the stream...but even that can't work; what about: $ (cat file-1; cat file-2) > file-3. Should the shell now know that when you run multiple commands, it should eat the BOM emitted from the second command? Basically, using a BOM in a utf-8 file is just not a good idea: it completely ruins interop with every standard unix tool. This is not to say that Python shouldn't have a way to read a file with a UTF-8 BOM: it just shouldn't encourage you to *write* such files. James From mal at egenix.com Fri Jan 8 22:51:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 08 Jan 2010 22:51:26 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> Message-ID: <4B47A8DE.1080401@egenix.com> Tres Seaver wrote: > M.-A. Lemburg wrote: > >> Shouldn't this encoding guessing be a separate function that you call >> on either a file or a seekable stream ? > >> After all, detecting encodings is just as useful to have for non-file >> streams. > > Other stream sources typically have out-of-band ways to signal the > encoding: only when reading from the filesystem do we pretty much > *have* to guess, and in that case the BOM / signature is the best > heuristic we have. Also, some non-file streams are not seekable, and so > can't be guessed via a pre-pass. Sure there are non-seekable file streams, but at least when using StringIO-type streams you don't have that restriction. An encoding detection function would provide more use in other cases as well, so instead of hiding away the functionality in the open() constructor, I'm suggesting to make expose it via the codecs module. Applications can then use it where necessary and also provide their own defaults, using other heuristics. >> You'd then avoid having to stuff everything into >> a single function call and also open up the door for more complex >> application specific guess work or defaults. > >> The whole process would then have two steps: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_file_encoding(filename) > > Filename is not enough information: or do you mean that API to actually > open the stream? Yes. The API would open the file, guess the encoding and then close it again. If you don't want that to happen, you could use the second API I mentioned below on the already open file. Note that this function could detect a lot more encodings with reasonably high probability than just BOM-prefixed ones, e.g. we could also add support to detect encoding declarations such as the ones we use in Python source files. >> 2. open the file with the found encoding > >> f = open(filename, encoding=encoding) > >> For seekable streams f, you'd have: > >> 1. guess encoding > >> import codecs >> encoding = codecs.guess_stream_encoding(f) I forgot to mention: This API needs to position the file pointer to the start of the first data byte. >> 2. wrap the stream with a reader for the found encoding > >> reader_class = codecs.getreader(encoding) >> g = reader_class(f) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Fri Jan 8 22:59:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 08 Jan 2010 16:59:04 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B475C72.1010207@egenix.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote: >>> Shouldn't this encoding guessing be a separate function that you call >>> on either a file or a seekable stream ? >>> >>> After all, detecting encodings is just as useful to have for non-file >>> streams. >> Other stream sources typically have out-of-band ways to signal the >> encoding: only when reading from the filesystem do we pretty much >> *have* to guess, and in that case the BOM / signature is the best >> heuristic we have. Also, some non-file streams are not seekable, and so >> can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). If the application expects a possibly-BOM-signature-marked file, but you pass it mismatched garbage: >>> f = open('some.zip', encoding='BOM") the error handling should be the same as if you passed any other mismatched encoding: >>> f = open('some.zip', encoding='UTF8') i.e., you discover the error when you try to read from the (non)encoded stream, not when you open it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktHqpwACgkQ+gerLs4ltQ7uAACeKEc+WT4TASGcVl1Hfqe6L9La I6EAn1pJtngtLWPdothGbYB+zUabEvTW =TjBK -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Fri Jan 8 23:10:32 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 8 Jan 2010 23:10:32 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <36707.63.251.87.214.1262986847.squirrel@mail.trueblade.com> Message-ID: <201001082310.33029.victor.stinner@haypocalc.com> Le vendredi 08 janvier 2010 22:40:47, Eric Smith a ?crit : > >> Shouldn't this encoding guessing be a separate function that you call > >> on either a file or a seekable stream ? > >> > >> After all, detecting encodings is just as useful to have for non-file > >> streams. > > > > Other stream sources typically have out-of-band ways to signal the > > encoding: only when reading from the filesystem do we pretty much > > *have* to guess, and in that case the BOM / signature is the best > > heuristic we have. Also, some non-file streams are not seekable, and so > > can't be guessed via a pre-pass. > > But what if the file were in (for example) a zip file? I think you > definitely want to have access to this functionality outside of open(). FYI my patch (encoding="BOM") is implemented in TextIOWrapper, and TextIOWrapper takes a binary stream as input, not a filename. -- Victor Stinner http://www.haypocalc.com/ From yoann.padioleau at facebook.com Fri Jan 8 23:59:52 2010 From: yoann.padioleau at facebook.com (Yoann Padioleau) Date: Fri, 8 Jan 2010 14:59:52 -0800 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: <4B464F1C.7020404@v.loewis.de> References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: On Jan 7, 2010, at 1:16 PM, Martin v. L?wis wrote: >>> astgen.py is not used to process asdl files; ast.txt lives right >>> next to astgen.py. Instead, the asdl file is processed by >>> Parser/asdl_c.py. >> >> Yes, I know that. That's why I asked about the relation between >> ast.txt and Python.adsl. If internally the parser uses the .adsl, but >> expose as a reflection mechanism things that were generated from >> ast.txt, then there could be a mismatch. Where does ast.txt comes >> from ? Shouldn't it be generated itself from Python.adsl ? > > What you may not be aware of is that Tools/compiler (and the > compiler package that it builds on) are both unused and unmaintained. I see. So if people want to analyze python code they have to use other tools (like rope?) ? or use reflection ? > > If the package stops working correctly - tough luck. > >> So we would have >> >> Python.adsl ----????----> ast.txt ---- astgen.py ---> ast.py >> containing all the UnarySub, Expression, classes that represents a >> Python AST. > > No - what actually happens in Python 3.x is this: both the compiler > package and Tools/compiler are removed. Ok. I will then create my own ast classes generator. Thanks. > > Regards, > Martin From g.brandl at gmx.net Sat Jan 9 00:10:24 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 09 Jan 2010 00:10:24 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F54D.9090103@v.loewis.de> Message-ID: Am 08.01.2010 22:14, schrieb Tres Seaver: >> FWIW, I'm personally in favor of using the UTF-8 signature. If people >> consider them crazy talk, that may be because UTF-8 can't possibly have >> a byte order - hence I call it a signature, not the BOM. As a signature, >> I don't consider it crazy at all. There is a long tradition of having >> magic bytes in files (executable files, Postscript, PDF, ... - see >> /etc/magic). Having a magic byte sequence for plain text to denote the >> encoding is useful and helps reducing moji-bake. This is the reason it's >> used on Windows: notepad would normally assume that text is in the ANSI >> code page, and for compatibility, it can't stop doing that. So the UTF-8 >> signature gives them an exit strategy. > > Agreed. Having that marker at the start of the file makes interop with > other tools *much* easier. Except if only 50% of the other tools support the signature. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jan 9 00:57:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 00:57:46 +0100 Subject: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt In-Reply-To: References: <7B83F217-4808-475E-95F2-FB64238C3ADD@facebook.com> <4B464499.4020305@v.loewis.de> <6A95699A-4770-4347-9BA8-2CCC853443B5@facebook.com> <4B464F1C.7020404@v.loewis.de> Message-ID: <4B47C67A.3020302@v.loewis.de> > I see. So if people want to analyze python code they have to use > other tools (like rope?) ? or use reflection ? Correct. One such tool might be the true Python compiler, along with the _ast module. Regards, Martin From victor.stinner at haypocalc.com Sat Jan 9 00:59:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 00:59:00 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM Message-ID: <201001090059.00848.victor.stinner@haypocalc.com> Hi, Thanks for all the answers! I will try to sum up all ideas here. (1) Change default open() behaviour or make it optional? Guido would like to add an option and keep open() unchanged. He wrote that checking for BOM and using system locale are too much different to be the same option (encoding=None). Antoine would like to check BOM by default, because both options (system locale vs checking for BOM) is the same thing. About Antoine choice (encoding=None): which file modes would check for a BOM? I would like to answer only the read only mode, but then open(filename, "r") and open(filename, "r+") would behave differently? => 1 point for Guido (encoding="BOM" is more explicit) Antoine choice has the advantage of directly support UTF-8+BOM, UTF-16 and UTF-32 for all applications and all modules using open(filename). => 1 point for Antoine (2) Check for a BOM while reading or detect it before? Everybody agree that checking BOM is an interesting option and should not be limited to open(). Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file name or a binary file-like object: it returns the encoding and seek to the file start or just after the BOM. I dislike this function because it requires extra file operations (open (optional), read() and seek()) and it doesn't work if the file is not seekable (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to avoid extra file operations. Note: I implemented the BOM check in TextIOWrapper; so it's already usable for any file-like object. (3) tell() and seek() on a text file starting with a BOM To be consistent with Antoine example: >>> bio = io.BytesIO(b'\xff\xfea\x00b\x00') >>> f = io.TextIOWrapper(bio, encoding='utf-16') >>> f.read() 'ab' >>> f.seek(0) 0 >>> f.read() 'ab' TextIOWrapper: * tell() should return zero at file start, * seek(0) should go be to file start, * and the BOM should always be "ignored". I mean: with open("utf8bom.txt", encoding="BOM") as fp: assert fp.tell() == 0 text = fp.read() # no BOM here fp.seek(0) assert fp.read() == text -- About my patch: - BOM check is explicit: open(filebame, encoding="BOM") - tell() / seek(0) works as expected - BOM bytes are always skipped in read() / readlines() result Hum, I don't know if this email can be called a sum up ;-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 01:09:48 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 00:09:48 +0000 (UTC) Subject: [Python-Dev] Quick sum up about open() + BOM References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: Hello Victor, Victor Stinner haypocalc.com> writes: > > (1) Change default open() behaviour or make it optional? > [...] > > Antoine would like to check BOM by default, because both options (system > locale vs checking for BOM) is the same thing. To be clear, I am not saying it is the same thing. What I think is that it would be a mistake to use a mildly unreliable heuristic by default (the locale + device encoding heuristic) but refuse to trust a more reliable heuristic (the BOM-based detection algorithm). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Jan 9 01:14:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 00:14:39 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47CA6F.5000607@voidspace.org.uk> On 09/01/2010 00:09, Antoine Pitrou wrote: > Hello Victor, > > Victor Stinner haypocalc.com> writes: > >> (1) Change default open() behaviour or make it optional? >> >> > [...] > >> Antoine would like to check BOM by default, because both options (system >> locale vs checking for BOM) is the same thing. >> > To be clear, I am not saying it is the same thing. What I think is that it would > be a mistake to use a mildly unreliable heuristic by default (the locale + > device encoding heuristic) but refuse to trust a more reliable heuristic (the > BOM-based detection algorithm). > I concur. On Windows both UTF-8 and signature are very common, yet the platform default is the truly awful CP1252. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From floris.bruynooghe at gmail.com Sat Jan 9 01:26:05 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 9 Jan 2010 00:26:05 +0000 Subject: [Python-Dev] --enabled-shared broken on freebsd5? In-Reply-To: <4B46F6D7.1050301@v.loewis.de> References: <66d0a6e11001061314k302b4e5xb5702cd6dc46a60e@mail.gmail.com> <4B450CF8.8090009@v.loewis.de> <66d0a6e11001061608u20fa89aeyed0c707452475cc9@mail.gmail.com> <66d0a6e11001071911v72da8326ued1221fdfda2e2ff@mail.gmail.com> <4B46F6D7.1050301@v.loewis.de> Message-ID: <20100109002605.GB13536@laurie.devork> On Fri, Jan 08, 2010 at 10:11:51AM +0100, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > I think this problem probably needs to move over to distutils-sig, as > > it doesn't seem to be specific to the way that Python itself uses > > distutils. > > I'm fairly skeptical that anybody on distutils SIG is interested in > details of the Python build process... Uh, hum. Unfounded skepticism. ;-) But as said filing a bug sounds better in this case. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From v+python at g.nevcal.com Sat Jan 9 01:47:38 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 16:47:38 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B47D22A.8070305@g.nevcal.com> On approximately 1/8/2010 3:59 PM, came the following characters from the keyboard of Victor Stinner: > Hi, > > Thanks for all the answers! I will try to sum up all ideas here. One concern I have with this implementation encoding="BOM" is that if there is no BOM it assumes UTF-8. That is probably a good assumption in some circumstances, but not in others. * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE encoded files include a BOM. It is only required that UTF-16 and UTF-32 (cases where the endianness is unspecified) contain a BOM. Hence, it might be that someone would expect a UTF-16LE (or any of the formats that don't require a BOM, rather than UTF-8), but be willing to accept any BOM-discriminated format. * Potentially, this could be expanded beyond the various Unicode encodings... one could envision that a program whose data files historically were in any particular national language locale, could want to be enhance to accept Unicode, and could declare that they will accept any BOM-discriminated format, but want to default, in the absence of a BOM, to the original national language locale that they historically accepted. That would provide a migration path for their old data files. So the point is, that it might be nice to have "BOM-otherEncodingForDefault" for each other encoding that Python supports. Not sure that is the right API, but I think it is expressive enough to handle the cases above. Whether the cases solve actual problems or not, I couldn't say, but they seem like reasonable cases. It would, of course, be nicest if OS metadata had been invented way back when, for all OSes, such that all text files were flagged with their encoding... then languages could just read the encoding and do the right thing! But we live in the real world, instead. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From python at mrabarnett.plus.com Sat Jan 9 02:12:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 09 Jan 2010 01:12:28 +0000 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B47D7FC.4070703@mrabarnett.plus.com> Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. > > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. > What about listing the possible encodings? It would try each in turn until it found one where the BOM matched or had no BOM: my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') or is that taking it too far? From martin at v.loewis.de Sat Jan 9 02:23:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Jan 2010 02:23:07 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47CA6F.5000607@voidspace.org.uk> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> Message-ID: <4B47DA7B.6050505@v.loewis.de> >>> Antoine would like to check BOM by default, because both options >>> (system locale vs checking for BOM) is the same thing. >>> >> To be clear, I am not saying it is the same thing. What I think is >> that it would be a mistake to use a mildly unreliable heuristic by >> default (the locale + device encoding heuristic) but refuse to >> trust a more reliable heuristic (the BOM-based detection >> algorithm). >> > > I concur. On Windows both UTF-8 and signature are very common, yet > the platform default is the truly awful CP1252. While I would support combining BOM detection in the case where a file is opened for reading and no encoding is specified, I see two problems: a) if a seek operations is performed before having looked at the BOM, no determination would have been made b) what encoding should it use on writing? Regards, Martin From v+python at g.nevcal.com Sat Jan 9 02:49:12 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 08 Jan 2010 17:49:12 -0800 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <4B47E098.6030703@g.nevcal.com> On approximately 1/8/2010 5:12 PM, came the following characters from the keyboard of MRAB: > Glenn Linderman wrote: >> On approximately 1/8/2010 3:59 PM, came the following characters from >> the keyboard of Victor Stinner: >>> Hi, >>> >>> Thanks for all the answers! I will try to sum up all ideas here. >> >> One concern I have with this implementation encoding="BOM" is that if >> there is no BOM it assumes UTF-8. That is probably a good assumption >> in some circumstances, but not in others. >> >> * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE >> encoded files include a BOM. It is only required that UTF-16 and >> UTF-32 (cases where the endianness is unspecified) contain a BOM. >> Hence, it might be that someone would expect a UTF-16LE (or any of >> the formats that don't require a BOM, rather than UTF-8), but be >> willing to accept any BOM-discriminated format. >> >> * Potentially, this could be expanded beyond the various Unicode >> encodings... one could envision that a program whose data files >> historically were in any particular national language locale, could >> want to be enhance to accept Unicode, and could declare that they >> will accept any BOM-discriminated format, but want to default, in the >> absence of a BOM, to the original national language locale that they >> historically accepted. That would provide a migration path for their >> old data files. >> >> So the point is, that it might be nice to have >> "BOM-otherEncodingForDefault" for each other encoding that Python >> supports. Not sure that is the right API, but I think it is >> expressive enough to handle the cases above. Whether the cases solve >> actual problems or not, I couldn't say, but they seem like reasonable >> cases. >> >> It would, of course, be nicest if OS metadata had been invented way >> back when, for all OSes, such that all text files were flagged with >> their encoding... then languages could just read the encoding and do >> the right thing! But we live in the real world, instead. >> > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? That sounds very flexible -- but in net effect it would only make illegal a subset of the BOM-containing encodings (those not listed) without making legal any additional encodings other than the non-BOM encoding. Whether prohibiting a subset of BOM-containing encodings is a useful use case, I couldn't say... but my goal would be to included as many different file encodings on input as possible: without a BOM, that is exactly 1 (unless there are other heuristics), with a BOM, it is 1+all-BOM-containing encodings. Your scheme would permit numbers of encodings accepted to vary between 1 and 1+all-BOM-containing encodings. (I think everyone can agree there are 5 different byte sequences that can be called a Unicode BOM. The likelihood of them appearing in any other text encoding created by mankind depends on those other encodings -- but it is not impossible. It is truly up to the application to decide whether BOM detection could potentially conflict with files in some other encoding that would be acceptable to the application.) So I think it is taking it further than I can see value in, but I'm willing to be convinced otherwise. I see only a need for detecting BOM, and specifying a default encoding to be used if there is no BOM. Note that it might be nice to have a specification for using current encoding=None heuristic -- perhaps encoding="BOM-None" per my originally proposed syntax. But I'm still not saying that is the best syntax. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ncoghlan at gmail.com Sat Jan 9 04:07:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 09 Jan 2010 13:07:12 +1000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B476196.4080409@mrabarnett.plus.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081127.44044.victor.stinner@haypocalc.com> <4B476196.4080409@mrabarnett.plus.com> Message-ID: <4B47F2E0.7090403@gmail.com> MRAB wrote: > Maybe there should also be a way of determining what encoding it decided > it was, so that you can then write a new file in that same encoding. I thought of that question as well - the f.encoding attribute on the opened file should be sufficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Sat Jan 9 06:48:36 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 06:48:36 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47E098.6030703@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> <4B47E098.6030703@g.nevcal.com> Message-ID: <319e029f1001082148h5cee2c0bn76f70a17766ca69e@mail.gmail.com> It seems to me that when opening a file, the following is the only flow that makes sense for the typical opening of a file flow: if encoding is not None: use encoding elif file has BOM: use BOM else: use system default And hence a encoding='BOM' isn't needed there. Although I'm trying to come up with usecases that doesn't work with this, I can't. :) BUT When writing things are not so easy though. Apparently some encodings require a BOM to be written, but others do not, but allow it, and some has no byte order mark. So there you have to be able to write the BOM, or not. And that's either a new parameter, because you can't use encoding='BOM' since you need to specify the encoding as well, or a new method. I would suggest a BOM parameter, and maybe a method as well. BOM=None|True|False Where "None" means a sane default behaviour, that is write a BOM if the encoding require it. "True" means write a BOM if the encoding *supports* it. "False" means Don't write a BOM even if the encoding requires it (because I know what I'm doing) if 'w' in mode: # But not 'r' or 'a' if BOM == True and encoding in (ENCODINGS THAT ALLOW BOM): write_bom = True elif BOM == False: write_bom = False elif BOM == None and encoding in (ENCODINGS THAT REQUIRE BOM): write_bom = True else: write_bom = False else: write_bom = False For reading this parameter could either be a noop, or possibly change the behavior somehow, if a usecase where that makes sense can be imagined. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From walter at livinglogic.de Sat Jan 9 11:51:57 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 11:51:57 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <4B485FCD.2040901@livinglogic.de> On 09.01.10 01:47, Glenn Linderman wrote: > On approximately 1/8/2010 3:59 PM, came the following characters from > the keyboard of Victor Stinner: >> Hi, >> >> Thanks for all the answers! I will try to sum up all ideas here. > > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. That is probably a good assumption in > some circumstances, but not in others. > > * It is not required that UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE > encoded files include a BOM. It is only required that UTF-16 and UTF-32 > (cases where the endianness is unspecified) contain a BOM. Hence, it > might be that someone would expect a UTF-16LE (or any of the formats > that don't require a BOM, rather than UTF-8), but be willing to accept > any BOM-discriminated format. > > * Potentially, this could be expanded beyond the various Unicode > encodings... one could envision that a program whose data files > historically were in any particular national language locale, could want > to be enhance to accept Unicode, and could declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted. That would provide a migration path for their old data files. > > So the point is, that it might be nice to have > "BOM-otherEncodingForDefault" for each other encoding that Python > supports. Not sure that is the right API, but I think it is expressive > enough to handle the cases above. Whether the cases solve actual > problems or not, I couldn't say, but they seem like reasonable cases. This is doable with the currect API. Simply define a codec search function that handles all encoding names that start with "BOM-" and pass the "otherEncodingForDefault" part along to the codec. > It would, of course, be nicest if OS metadata had been invented way back > when, for all OSes, such that all text files were flagged with their > encoding... then languages could just read the encoding and do the right > thing! But we live in the real world, instead. Servus, Walter From walter at livinglogic.de Sat Jan 9 12:18:33 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 09 Jan 2010 12:18:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001081140.28124.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> Message-ID: <4B486609.2050804@livinglogic.de> Victor Stinner wrote: > Le vendredi 08 janvier 2010 10:10:23, Martin v. L?wis a ?crit : >>> Builtin open() function is unable to open an UTF-16/32 file starting with >>> a BOM if the encoding is not specified (raise an unicode error). For an >>> UTF-8 file starting with a BOM, read()/readline() returns also the BOM >>> whereas the BOM should be "ignored". >> It depends. If you use the utf-8-sig encoding, it *will* ignore the >> UTF-8 signature. > > Sure, but it means that you only use UTF-8+BOM files. If you get UTF-8 and > UTF-8+BOM files, you have to to detect the encoding (not an easy job) or to > remove the BOM after the first read (much harder if you use a module like > ConfigParser or csv). > >>> Since my proposition changes the result TextIOWrapper.read()/readline() >>> for files starting with a BOM, we might introduce an option to open() to >>> enable the new behaviour. But is it really needed to keep the backward >>> compatibility? >> Absolutely. And there is no need to produce a new option, but instead >> use the existing options: define an encoding that auto-detects the >> encoding from the family of BOMs. Maybe you call it encoding="sniff". > > Good idea, I choosed open(filename, encoding="BOM"). On the surface this looks like there's an encoding named "BOM", but looking at your patch I found that the check is still done in TextIOWrapper. IMHO the best approach would to the implement a *real* codec named "BOM" (or "sniff"). This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop. To see how something like this can be done, take a look at the UTF-16 codec, that switches to bigendian or littleendian mode depending on the first read/decode call. Servus, Walter From victor.stinner at haypocalc.com Sat Jan 9 13:37:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:37:06 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47DA7B.6050505@v.loewis.de> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47CA6F.5000607@voidspace.org.uk> <4B47DA7B.6050505@v.loewis.de> Message-ID: <201001091337.06596.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:23:07, Martin v. L?wis a ?crit : > While I would support combining BOM detection in the case where a file > is opened for reading and no encoding is specified, I see two problems: > a) if a seek operations is performed before having looked at the BOM, > no determination would have been made TextIOWrapper doesn't support seek to an arbitrary byte. It uses "cookie" which is an opaque value. Reuse a cookie from another file or an old cookie is forbidden (but it doesn't raise an error). This is not specific to the BOM checking: the problem already exist for encodings using a BOM (eg. UTF-16). > b) what encoding should it use on writing? Don't change anything to writing. With Antoince choice: open('file.txt', 'w', encoding=None) continue to use the actual heuristic (os.device_encoding() or system locale). With Guido choice, encoding="BOM": it raises an error, because BOM check is not supported when writing into a file. How could the BOM be checked when creating a new (empty) file!? -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Sat Jan 9 13:45:58 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 09 Jan 2010 13:45:58 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <201001090059.00848.victor.stinner@haypocalc.com> References: <201001090059.00848.victor.stinner@haypocalc.com> Message-ID: <4B487A86.4020603@egenix.com> Victor Stinner wrote: > (2) Check for a BOM while reading or detect it before? > > Everybody agree that checking BOM is an interesting option and should not be > limited to open(). > > Marc-Andre proposed a codecs.guess_file_encoding() function accepting a file > name or a binary file-like object: it returns the encoding and seek to the > file start or just after the BOM. > > I dislike this function because it requires extra file operations (open > (optional), read() and seek()) and it doesn't work if the file is not seekable > (eg. a pipe). I prefer to check for a BOM at first read in TextIOWrapper to > avoid extra file operations. > > Note: I implemented the BOM check in TextIOWrapper; so it's already usable for > any file-like object. Yes, but the implementation is limited to just BOM checking and thus only supports UTF-8-SIG, UTF-16 and UTF-32. With a codecs module function we could easily extend the encoding detection to more file types, e.g. XML files, Python source code files, etc. that use other mechanisms for defining the encoding. BTW: I haven't looked at your implementation, but what happens when your BOM check fails ? Will the implementation add the already read bytes back to a buffer ? This rollback action is the only reason for needing a seekable stream in codecs.guess_stream_encoding(). Another point to consider: AFAIK, we currently have a moratorium on changes to Python builtins. How does that match up with the proposed changes ? Using a new codec like Walter suggested would move the implementation into the stdlib for which doesn't the moratorium doesn't apply. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Sat Jan 9 13:54:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 13:54:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D22A.8070305@g.nevcal.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> Message-ID: <201001091354.17239.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 01:47:38, vous avez ?crit : > One concern I have with this implementation encoding="BOM" is that if > there is no BOM it assumes UTF-8. If no BOM is found, it fallback to the current heuristic: os.device_encoding() or system local. > (...) Hence, it might be that someone would expect a UTF-16LE (or any of > the formats that don't require a BOM, rather than UTF-8), but be willing > to accept any BOM-discriminated format. > (...) declare that they will accept > any BOM-discriminated format, but want to default, in the absence of a > BOM, to the original national language locale that they historically > accepted You mean "if there is a BOM, use it, otherwise fallback to a specific charset"? How could it be declared? Maybe: open("file.txt", check_bom=True, encoding="UTF16-LE") open("file.txt", check_bom=True, encoding="latin1") About falling back to UTF-8, it would be written: open("file.txt", check_bom=True, encoding="UTF-8") As explained before, check_bom=True is only accepted for read only file mode. Well, why not. This is a third choice for my point (1) :-) It's between Guido and Antoine choice, and I like it because we can fallback to UTF-8 instead of the dummy system locale: Windows users will be happy to be able to use UTF-8 :-) I prefer to fallback to a fixed encoding then depending on the system locale. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:34:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:34:17 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B47D7FC.4070703@mrabarnett.plus.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B47D22A.8070305@g.nevcal.com> <4B47D7FC.4070703@mrabarnett.plus.com> Message-ID: <201001091434.17521.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 02:12:28, MRAB a ?crit : > What about listing the possible encodings? It would try each in turn > until it found one where the BOM matched or had no BOM: > > my_file = open(filename, 'r', encoding='UTF-8-sig|UTF-16|UTF-8') > > or is that taking it too far? Yes, you're taking it foo far :-) Checking BOM is reliable, whereas *guessing* the charset only using the byte stream can only be an heuristic. Guess a charset is a complex problem, they are 3rd party library to do that, like the chardet project. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:38:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:38:43 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B486609.2050804@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <201001091438.43576.victor.stinner@haypocalc.com> Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : > > Good idea, I choosed open(filename, encoding="BOM"). > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Why not, this is another solution to the point (2) (Check for a BOM while reading or detect it before?). Which encoding would be used if there is not BOM? UTF-8 sounds like a good choice. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Sat Jan 9 14:50:28 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 9 Jan 2010 14:50:28 +0100 Subject: [Python-Dev] Quick sum up about open() + BOM In-Reply-To: <4B487A86.4020603@egenix.com> References: <201001090059.00848.victor.stinner@haypocalc.com> <4B487A86.4020603@egenix.com> Message-ID: <201001091450.28497.victor.stinner@haypocalc.com> Hi, Le samedi 09 janvier 2010 13:45:58, vous avez ?crit : > > Note: I implemented the BOM check in TextIOWrapper; so it's already > > usable for any file-like object. > > Yes, but the implementation is limited to just BOM checking > and thus only supports UTF-8-SIG, UTF-16 and UTF-32. Sure, but that's already better than no BOM check :-) It looks like many people would apprecite UTF-8-SIG detection, since this encoding is common on Windows. > BTW: I haven't looked at your implementation, but what happens > when your BOM check fails ? Will the implementation add the > already read bytes back to a buffer ? My implementation is done between buffer.read() and decoder.decode(data). If there is a BOM: set the encoding and remove the BOM bytes from the byte string. Otherwise, use another algorithm to choose the encoding and leave the byte string unchanged. It can be seen as a codec: it works like UTF-16 and UTF-32 codecs ;-) > AFAIK, we currently have a moratorium on changes to Python > builtins. How does that match up with the proposed changes ? Oh yes, I forgot the moratorium. In all solutions, some of them don't change the API. Eg. Antoine proposed to leave the API unchanged: open(file) => open(file) :-) I don't know if it's compatible with the moratorium or not. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Sat Jan 9 16:05:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 15:05:45 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: Walter D?rwald livinglogic.de> writes: > > On the surface this looks like there's an encoding named "BOM", but > looking at your patch I found that the check is still done in > TextIOWrapper. IMHO the best approach would to the implement a *real* > codec named "BOM" (or "sniff"). This doesn't require *any* changes to > the IO library. It could even be developed as a standalone project and > published in the Cheeseshop. Sorry but this is missing the point. The point here is to improve the open() function. I'm sure people who know about encodings are able to install the chardet library or even whip up their own BOM detection routine... Antoine. From benjamin at python.org Sat Jan 9 18:29:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 11:29:33 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 Message-ID: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> On behalf of the Python development team, I'm gleeful to announce the second alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From kmtracey at gmail.com Sat Jan 9 19:48:12 2010 From: kmtracey at gmail.com (Karen Tracey) Date: Sat, 9 Jan 2010 13:48:12 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce the > second > alpha release of Python 2.7. > > Well yay. Django's test suite (1242 tests) runs with just one failure on the 2.7 alpha 2 level, and that looks to be likely due to the improved string/float rounding so not really a problem, just a difference. That's down from 104 failures and 40 errors with 2.7 alpha 1. Note on the website page http://www.python.org/download/releases/2.7/ the "Change log for this release" link is still pointing to the alpha 1 changelog. Thanks, Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jan 9 19:51:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Jan 2010 12:51:11 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <1afaf6161001091051kb1ba08q9b96094af16c12fa@mail.gmail.com> 2010/1/9 Karen Tracey : > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson > wrote: >> >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> > > Well yay.? Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference.? That's > down from 104 failures and 40 errors with 2.7 alpha 1. Excellent! > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Thanks. I'll fix that. > -- Regards, Benjamin From skip at pobox.com Sat Jan 9 21:00:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:00:44 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 Message-ID: <19272.57452.972234.643008@montanaro.dyndns.org> How much of the Unladen Swallow cPickle speedups have been incorporated into 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the two versions I currently care about at work - we will skip 2.5 entirely). It appears some of their speedups may have already been merged to trunk, but I'm not sure how much. If a patch to merge this to 2.7 is already under consideration I won't look at it, but I interpreted Collin Winter's response to my query on the u-s mailing list that not everything has been done yet. Thx, Skip From martin at v.loewis.de Sat Jan 9 21:09:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 09 Jan 2010 21:09:27 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> Message-ID: <4B48E277.9010408@v.loewis.de> Antoine Pitrou wrote: > Walter D?rwald livinglogic.de> writes: >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Sorry but this is missing the point. The point here is to improve the open() > function. I'm sure people who know about encodings are able to install the > chardet library or even whip up their own BOM detection routine... How does the requirement that it be implemented as a codec miss the point? FWIW, I agree with Walter that if it is provided through the encoding= argument, it should be a codec. If it is built into the open function (for whatever reason), it must be provided by some other parameter. I do see the point that it becomes available to end users only when released as part of Python. However, this *also* means that applications won't be using it for another three years or so, since they'll have to support older Python versions as well (unless it is integrated in the case where no encoding is specified). Regards, Martin From pjenvey at underboss.org Sat Jan 9 21:09:29 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 9 Jan 2010 12:09:29 -0800 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <19272.57452.972234.643008@montanaro.dyndns.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> On Jan 9, 2010, at 12:00 PM, skip at pobox.com wrote: > How much of the Unladen Swallow cPickle speedups have been incorporated into > 2.7 & 3.1? I'm working on trying to develop patches for 2.4 and 2.6 (the > two versions I currently care about at work - we will skip 2.5 entirely). > It appears some of their speedups may have already been merged to trunk, but > I'm not sure how much. If a patch to merge this to 2.7 is already under > consideration I won't look at it, but I interpreted Collin Winter's response > to my query on the u-s mailing list that not everything has been done yet. They've documented their upstream patches here: http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches -- Philip Jenvey From skip at pobox.com Sat Jan 9 21:21:20 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:21:20 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> References: <19272.57452.972234.643008@montanaro.dyndns.org> <7855D115-33BE-47F6-BDA9-ABC0A4D7E759@underboss.org> Message-ID: <19272.58688.173011.412712@montanaro.dyndns.org> Philip> They've documented their upstream patches here: Philip> http://code.google.com/p/unladen-swallow/wiki/UpstreamPatches Thanks. That will help immensely. Skip From solipsis at pitrou.net Sat Jan 9 21:28:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:28:12 +0000 (UTC) Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Sorry but this is missing the point. The point here is to improve the open() > > function. I'm sure people who know about encodings are able to install the > > chardet library or even whip up their own BOM detection routine... > > How does the requirement that it be implemented as a codec miss the > point? If we want it to be the default, it must be able to fallback on the current locale-based algorithm if no BOM is found. I don't think it would be easy for a codec to do that. > FWIW, I agree with Walter that if it is provided through the encoding= > argument, it should be a codec. If it is built into the open function > (for whatever reason), it must be provided by some other parameter. Why not simply encoding=None? The default value should provide the most useful behaviour possible. Forcing users to choose between two different autodetection strategies (encoding=None and another one) is a little insane IMO. Regards Antoine. From solipsis at pitrou.net Sat Jan 9 21:32:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 9 Jan 2010 20:32:27 +0000 (UTC) Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > If a patch to merge this to 2.7 is already under > consideration I won't look at it, Why won't you look at it? :) Actually, if these patches are to be merged someone should certainly look at them, and do the (possibly) remaining work. http://bugs.python.org/issue5683 http://bugs.python.org/issue5671 Thank you Antoine. From skip at pobox.com Sat Jan 9 21:43:42 2010 From: skip at pobox.com (skip at pobox.com) Date: Sat, 9 Jan 2010 14:43:42 -0600 Subject: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1 In-Reply-To: References: <19272.57452.972234.643008@montanaro.dyndns.org> Message-ID: <19272.60030.25319.269597@montanaro.dyndns.org> >>>>> "Antoine" == Antoine Pitrou writes: Antoine> pobox.com> writes: >> >> If a patch to merge this to 2.7 is already under >> consideration I won't look at it, Antoine> Why won't you look at it? :) I meant I wouldn't look at developing one. Skip From regebro at gmail.com Sat Jan 9 23:14:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 9 Jan 2010 23:14:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Right. It seems like encoding=None is the right way to go there. encoding='BOM' would probably only work if 'BOM' isn't an encoding but a special tag, which is ugly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Sun Jan 10 00:25:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 09 Jan 2010 23:25:18 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <319e029f1001091414w7cb5e296w5c5e38e1a23ab3d5@mail.gmail.com> Message-ID: <4B49105E.303@voidspace.org.uk> On 09/01/2010 22:14, Lennart Regebro wrote: > On Sat, Jan 9, 2010 at 21:28, Antoine Pitrou wrote: > >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. >> > Right. It seems like encoding=None is the right way to go there. > encoding='BOM' would probably only work if 'BOM' isn't an encoding but > a special tag, which is ugly. > > I would rather see it as the default behavior for open without an encoding specified. I know Guido has expressed a preference against this so I won't continue to flog it. The current behavior however is that we have a 'guessing' algorithm based on the platform default. Currently if you open a text file in read mode that has a UTF-8 signature, but the platform default is something other than UTF-8, then we open the file using what is likely to be the incorrect encoding. Looking for the signature seems to be better behaviour in that case. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jan 10 00:40:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 10 Jan 2010 00:40:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> Message-ID: <4B4913DF.7050801@v.loewis.de> >> How does the requirement that it be implemented as a codec miss the >> point? > > If we want it to be the default, it must be able to fallback on the current > locale-based algorithm if no BOM is found. I don't think it would be easy for a > codec to do that. Yes - however, Victor currently apparently *doesn't* want it to be the default, but wants the user to specify encoding="BOM". If so, it isn't the default, and it is easy to implement as a codec. >> FWIW, I agree with Walter that if it is provided through the encoding= >> argument, it should be a codec. If it is built into the open function >> (for whatever reason), it must be provided by some other parameter. > > Why not simply encoding=None? I don't mind. Please re-read Walter's message - it only said that *if* this is activated through encoding="BOM", *then* it must be a codec, and could be on PyPI. I don't think Walter was talking about the case "it is not activated through encoding='BOM'" *at all*. > The default value should provide the most useful > behaviour possible. Forcing users to choose between two different autodetection > strategies (encoding=None and another one) is a little insane IMO. That wouldn't disturb me much. There are a lot of things in that area that are a little insane, starting with Microsoft Windows :-) Regards, Martin From henning.vonbargen at arcor.de Sun Jan 10 12:10:02 2010 From: henning.vonbargen at arcor.de (Henning von Bargen) Date: Sun, 10 Jan 2010 12:10:02 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: Message-ID: <4B49B58A.4040103@arcor.de> If Python should support BOM when reading text files, it should also be able to *write* such files. An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what? That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values: None: default (old) behaviour: don't handle BOM at all True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument. (*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False. Henning P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway... From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Sun Jan 10 12:21:48 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Jan 2010 12:21:48 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B49B58A.4040103@arcor.de> References: <4B49B58A.4040103@arcor.de> Message-ID: <319e029f1001100321pfed5deatd7483a49be343ba7@mail.gmail.com> On Sun, Jan 10, 2010 at 12:10, Henning von Bargen wrote: > If Python should support BOM when reading text files, > it should also be able to *write* such files. That's what I thought too. Turns out the UTF-16 does write such a mark. You also have the constants in the codecs module, so you can write the utf-16-le BOM and then use the utf-16-le encoding if you want to be sure you write utf-16-le, and the same with BE, of course. I still think now using BOM's when determining the file format can be seen as a bug, though, so I don't think the API needs to change at all. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brett at python.org Sun Jan 10 19:51:26 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 10:51:26 -0800 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: Nick Coghlan thought I should forward this to python-dev so people are aware of this change and why it occurred (plus it indirectly informs Guido I finally finished the work). ---------- Forwarded message ---------- From: brett.cannon Date: Sat, Jan 9, 2010 at 18:56 Subject: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c To: python-checkins at python.org Author: brett.cannon Date: Sun Jan 10 03:56:19 2010 New Revision: 77402 Log: DeprecationWarning is now silent by default. This was originally suggested by Guido, discussed on the stdlib-sig mailing list, and given the OK by Guido directly to me. What this change essentially means is that Python has taken a policy of silencing warnings that are only of interest to developers by default. This should prevent users from seeing warnings which are triggered by an application being run against a new interpreter before the app developer has a chance to update their code. Closes issue #7319. Thanks to Antoine Pitrou, Ezio Melotti, and Brian Curtin for helping with the issue. Modified: python/trunk/Doc/library/warnings.rst python/trunk/Lib/test/test_ascii_formatd.py python/trunk/Lib/test/test_exceptions.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS python/trunk/Python/_warnings.c Modified: python/trunk/Doc/library/warnings.rst ============================================================================== --- python/trunk/Doc/library/warnings.rst (original) +++ python/trunk/Doc/library/warnings.rst Sun Jan 10 03:56:19 2010 @@ -59,7 +59,7 @@ | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,14 +151,6 @@ :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - .. _warning-suppress: @@ -226,6 +221,37 @@ entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions Modified: python/trunk/Lib/test/test_ascii_formatd.py ============================================================================== --- python/trunk/Lib/test/test_ascii_formatd.py (original) +++ python/trunk/Lib/test/test_ascii_formatd.py Sun Jan 10 03:56:19 2010 @@ -4,6 +4,7 @@ import unittest from test_support import check_warnings, run_unittest, cpython_only +import warnings class FormatDeprecationTests(unittest.TestCase): @@ -17,6 +18,7 @@ buf = create_string_buffer(' ' * 100) with check_warnings() as w: + warnings.simplefilter('default') PyOS_ascii_formatd(byref(buf), sizeof(buf), '%+.10f', c_double(10.0)) self.assertEqual(buf.value, '+10.0000000000') Modified: python/trunk/Lib/test/test_exceptions.py ============================================================================== --- python/trunk/Lib/test/test_exceptions.py (original) +++ python/trunk/Lib/test/test_exceptions.py Sun Jan 10 03:56:19 2010 @@ -309,6 +309,7 @@ # BaseException.__init__ triggers a deprecation warning. exc = BaseException("foo") with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('default') self.assertEquals(exc.message, "foo") self.assertEquals(len(w), 1) self.assertEquals(w[0].category, DeprecationWarning) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Sun Jan 10 03:56:19 2010 @@ -383,8 +383,8 @@ # Module initialization _processoptions(sys.warnoptions) if not _warnings_defaults: - simplefilter("ignore", category=PendingDeprecationWarning, append=1) - simplefilter("ignore", category=ImportWarning, append=1) + for cls in (DeprecationWarning, PendingDeprecationWarning, ImportWarning): + simplefilter("ignore", category=cls, append=True) bytes_warning = sys.flags.bytes_warning if bytes_warning > 1: bytes_action = "error" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Jan 10 03:56:19 2010 @@ -12,6 +12,8 @@ Core and Builtins ----------------- +- Issue #7319: Silence DeprecationWarning by default. + - Issue #2335: Backport set literals syntax from Python 3.x. Library Modified: python/trunk/Python/_warnings.c ============================================================================== --- python/trunk/Python/_warnings.c (original) +++ python/trunk/Python/_warnings.c Sun Jan 10 03:56:19 2010 @@ -85,10 +85,10 @@ default_action = get_warnings_attr("defaultaction"); if (default_action == NULL) { - if (PyErr_Occurred()) { - return NULL; - } - return _default_action; + if (PyErr_Occurred()) { + return NULL; + } + return _default_action; } Py_DECREF(_default_action); @@ -202,12 +202,12 @@ mod_str = PyString_AsString(filename); if (mod_str == NULL) - return NULL; + return NULL; len = PyString_Size(filename); if (len < 0) return NULL; if (len >= 3 && - strncmp(mod_str + (len - 3), ".py", 3) == 0) { + strncmp(mod_str + (len - 3), ".py", 3) == 0) { module = PyString_FromStringAndSize(mod_str, len-3); } else { @@ -251,7 +251,7 @@ name = PyObject_GetAttrString(category, "__name__"); if (name == NULL) /* XXX Can an object lack a '__name__' attribute? */ - return; + return; f_stderr = PySys_GetObject("stderr"); if (f_stderr == NULL) { @@ -341,7 +341,7 @@ rc = already_warned(registry, key, 0); if (rc == -1) goto cleanup; - else if (rc == 1) + else if (rc == 1) goto return_none; /* Else this warning hasn't been generated before. */ } @@ -492,9 +492,9 @@ /* Setup filename. */ *filename = PyDict_GetItemString(globals, "__file__"); if (*filename != NULL) { - Py_ssize_t len = PyString_Size(*filename); + Py_ssize_t len = PyString_Size(*filename); const char *file_str = PyString_AsString(*filename); - if (file_str == NULL || (len < 0 && PyErr_Occurred())) + if (file_str == NULL || (len < 0 && PyErr_Occurred())) goto handle_error; /* if filename.lower().endswith((".pyc", ".pyo")): */ @@ -506,10 +506,10 @@ tolower(file_str[len-1]) == 'o')) { *filename = PyString_FromStringAndSize(file_str, len-1); - if (*filename == NULL) - goto handle_error; - } - else + if (*filename == NULL) + goto handle_error; + } + else Py_INCREF(*filename); } else { @@ -536,8 +536,8 @@ else { /* embedded interpreters don't have sys.argv, see bug #839151 */ *filename = PyString_FromString("__main__"); - if (*filename == NULL) - goto handle_error; + if (*filename == NULL) + goto handle_error; } } if (*filename == NULL) { @@ -839,26 +839,29 @@ static PyObject * init_filters(void) { - PyObject *filters = PyList_New(3); + PyObject *filters = PyList_New(4); const char *bytes_action; if (filters == NULL) return NULL; PyList_SET_ITEM(filters, 0, + create_filter(PyExc_DeprecationWarning, "ignore")); + PyList_SET_ITEM(filters, 1, create_filter(PyExc_PendingDeprecationWarning, "ignore")); - PyList_SET_ITEM(filters, 1, create_filter(PyExc_ImportWarning, "ignore")); + PyList_SET_ITEM(filters, 2, create_filter(PyExc_ImportWarning, "ignore")); if (Py_BytesWarningFlag > 1) bytes_action = "error"; else if (Py_BytesWarningFlag) bytes_action = "default"; else bytes_action = "ignore"; - PyList_SET_ITEM(filters, 2, create_filter(PyExc_BytesWarning, + PyList_SET_ITEM(filters, 3, create_filter(PyExc_BytesWarning, bytes_action)); if (PyList_GET_ITEM(filters, 0) == NULL || PyList_GET_ITEM(filters, 1) == NULL || - PyList_GET_ITEM(filters, 2) == NULL) { + PyList_GET_ITEM(filters, 2) == NULL || + PyList_GET_ITEM(filters, 3) == NULL) { Py_DECREF(filters); return NULL; } _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sun Jan 10 20:22:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 10 Jan 2010 20:22:10 +0100 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> Message-ID: <201001102022.10259.victor.stinner@haypocalc.com> Hi, Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : > Nick Coghlan thought I should forward this to python-dev so people are > aware of this change and why it occurred (plus it indirectly informs Guido > I finally finished the work). Thanks to copy this mail to the python-dev mailing list. What is the recommanded way to get the previous behaviour (print deprecation warnings once)? Is there a command line option? Is it "-W default"? -- Victor Stinner http://www.haypocalc.com/ From benjamin at python.org Sun Jan 10 20:23:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 13:23:54 -0600 Subject: [Python-Dev] Fwd: [Python-checkins] r77402 - in python/trunk: Doc/library/warnings.rst Lib/test/test_ascii_formatd.py Lib/test/test_exceptions.py Lib/warnings.py Misc/NEWS Python/_warnings.c In-Reply-To: <201001102022.10259.victor.stinner@haypocalc.com> References: <4b4941df.0967f10a.71e8.6c41SMTPIN_ADDED@mx.google.com> <201001102022.10259.victor.stinner@haypocalc.com> Message-ID: <1afaf6161001101123g366d80dbjeaa80f1a632605e2@mail.gmail.com> 2010/1/10 Victor Stinner : > Hi, > > Le dimanche 10 janvier 2010 19:51:26, Brett Cannon a ?crit : >> Nick Coghlan thought I should forward this to python-dev so people are >> ?aware of this change and why it occurred (plus it indirectly informs Guido >> ?I finally finished the work). > > Thanks to copy this mail to the python-dev mailing list. > > What is the recommanded way to get the previous behaviour (print deprecation > warnings once)? Is there a command line option? Is it "-W default"? That commit conveniently adds documentation answering that question. :) -- Regards, Benjamin From nas at arctrix.com Sun Jan 10 20:30:09 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:30:09 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On behalf of the Python development team, I'm gleeful to announce > the second alpha release of Python 2.7. Thanks to everyone who contributed. > Python 2.7 is scheduled to be the last major version in the 2.x > series. Has this really been decided already? Maybe I missed it. In my opinion, it does Python's reputation harm to make such a statement. Conservative users (probably the vast majority of Python users) don't like to hear that software they are considering using is nearing the end of its life. What does it gain us to announce that the 2.x branch is dead aside from bugfixes? I propose that the 2.x branch be treated like 2.x.y branches: as long as there is sufficient volunteer labour, it should continue to live. In order to avoid wasted development effort, it would be prudent to announce that unless a 2.8 release manager steps up, whatever is committed to the 2.x branch after the 2.7 release may never get released. Said another way, it's okay for the Python developers to decide to abandon 2.x and put their efforts into 3.x. It's not okay for them to prevent others from continuing to work on 2.x or to somehow make 2.x look worse so 3.x looks better. Python 3 needs to stand on its own terms and I'm confident it can. Best regards, Neil From brett at python.org Sun Jan 10 21:09:08 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:09:08 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sun, Jan 10, 2010 at 11:30, Neil Schemenauer wrote: > Benjamin Peterson wrote: > > On behalf of the Python development team, I'm gleeful to announce > > the second alpha release of Python 2.7. > > Thanks to everyone who contributed. > > > Python 2.7 is scheduled to be the last major version in the 2.x > > series. > > Has this really been decided already? Maybe I missed it. More or less. It was first discussed at the language summit last year and has come up here a couple of times. If needed we can make it official in terms of lifetime of 2.7, etc. at the language summit this year. > In my > opinion, it does Python's reputation harm to make such a statement. > Conservative users (probably the vast majority of Python users) > don't like to hear that software they are considering using is > nearing the end of its life. What does it gain us to announce that > the 2.x branch is dead aside from bugfixes? > > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. > > Said another way, it's okay for the Python developers to decide to > abandon 2.x and put their efforts into 3.x. It's not okay for them > to prevent others from continuing to work on 2.x or to somehow make > 2.x look worse so 3.x looks better. Python 3 needs to stand on its > own terms and I'm confident it can. > I don't think ending the 2.x series at 2.7 makes it look bad compared to 3.2; it's simply the end of a development line like any other software project. I suspect 2.7 will have a protracted bugfix window because so much code runs on 2.x exclusively at the moment. And if core developers want to continue to backport fixes past two years from release they can (or however long we decide to officially support 2.7). No one is saying people still can't work on the code, just that python-dev as an entity is not going to focus its effort on the 2.x series anymore and people should not rely upon us to continue to focus new development efforts in that series. If there are core developers who want to continue to do bugfix releases then that's fine and I am happy to flag patches as needing backports and let other's do the work after the standard two year maintenance cycle, but I know I do not want to be held accountable as a core developer to keep the 2.x going indefinitely. Maintaining four branches is enough of a reason in my book to not keep the 2.x series going. If there really is an outcry on this we can re-visit the issue, but as of right now we need to move forward at some point and 2.7 seems like that good point. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 10 21:23:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Jan 2010 06:23:27 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A373F.9050601@gmail.com> Neil Schemenauer wrote: > In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. The announcement is precisely to avoid the situation where people commit new features to the 2.x main line of development (after the 2.7 maintenance branch is created) in the expectation that they will be released as part of a hypothetical 2.8 release. Whether that info needs to be in each and every 2.7 announcement... probably not. It isn't really info for users of Python, just for developers of Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jan 10 21:25:29 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 12:25:29 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit Message-ID: Michael has given me the hg transition/stdlib time slot at the language summit this year. In regards to that I plan to lead a discussion on: * where we are at w/ the Hg transition (Dirkjan should be there and I did a blog post on this topic recently: http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) * argparse (PEP 389) * brief mention on still wanting to break out the stdlib from CPython * an official policy on extension modules? (i.e. must have a pure Python implementation which can mean a ctypes implementation unless given an explicit waiver) That's everything from a stdlib perspective. I have half-baked ideas, but nothing concrete enough to bring up unless people really want to discuss stuff like how to potentially standardize module deprecation warnings, etc. In terms of me personally, I do plan to bring up at some point during the summit these points which don't squarely fit during my time slot: * an official unofficial policy on how new proposed features should come to us (i.e. working code to python-ideas with a shell of a PEP that includes abstract and motivation -> hashed out PEP to python-dev -> pronouncement) * any changes needed to the issue tracker to help with the workflow? (stage field seems like a failed experiment and we now have several effective triage people who can help w/ guiding changes) If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Sun Jan 10 22:52:00 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 10 Jan 2010 22:52:00 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) Unfortunately my flight doesn't land until about the time the Mercurial slot ends, so while I'll be there later on that afternoon for sprinting or questions or anything, I won't be at the actual Mercurial talk at the summit. Cheers, Dirkjan From fuzzyman at voidspace.org.uk Sun Jan 10 23:27:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 10 Jan 2010 22:27:58 +0000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4A546E.8010808@voidspace.org.uk> On 10/01/2010 21:52, Dirkjan Ochtman wrote: > On Sun, Jan 10, 2010 at 21:25, Brett Cannon wrote: > >> Michael has given me the hg transition/stdlib time slot at the language >> summit this year. In regards to that I plan to lead a discussion on: >> * where we are at w/ the Hg transition (Dirkjan should be there and I did a >> blog post on this topic recently: >> http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) >> > Unfortunately my flight doesn't land until about the time the > Mercurial slot ends, so while I'll be there later on that afternoon > for sprinting or questions or anything, I won't be at the actual > Mercurial talk at the summit. > > We will probably be in 'open discussion' when you arrive so it will still be good to hear from you on the topic and there maybe questions that can't be answered until you arrive... :-) Michael > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Mon Jan 11 00:02:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:02:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4A5C69.7090007@v.loewis.de> > I propose that the 2.x branch be treated like 2.x.y branches: as > long as there is sufficient volunteer labour, it should continue to > live. In order to avoid wasted development effort, it would be > prudent to announce that unless a 2.8 release manager steps up, > whatever is committed to the 2.x branch after the 2.7 release may > never get released. I think it's more difficult than that. It also takes developer effort to *commit* to the trunk. So if the trunk continued to live, would it still be ok if I closed a 2.x feature request as "won't fix", or only committed the new feature to the 3.x branch? As for old branches: they *don't* live in the way you claim (i.e. remain open with changes potentially just not released). Instead, at some point, they are frozen to bug-fix only, then to security-fix only, and then to no change at all. Regards, Martin From martin at v.loewis.de Mon Jan 11 00:07:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 00:07:58 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A373F.9050601@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> Message-ID: <4B4A5DCE.3070909@v.loewis.de> > The announcement is precisely to avoid the situation where people commit > new features to the 2.x main line of development (after the 2.7 > maintenance branch is created) in the expectation that they will be > released as part of a hypothetical 2.8 release. > > Whether that info needs to be in each and every 2.7 announcement... > probably not. It isn't really info for users of Python, just for > developers of Python. I think the announcement is indeed also for the users, and I would prefer it to stay in the release announcements, so that people will know for sure (and speak up) before the 2.7 release happens. As for decisions: I don't think there was an official BDFL pronouncent, but I recall Guido posting a message close to that, proposing that 2.7 will be a release that will see bug fix releases for an indefinite period of time (where indefinite != infinite). This was shortly after him proposing that perhaps we shouldn't make a 2.7 release at all, and stop at 2.6. As for such a decision giving a bad light on Python: I don't think that will be the case. Instead, I hear many users surprised for how long we have been maintaining to parallel versions - "that must have taken a lot of effort". So everybody will likely understand that enough is enough. Regards, Martin From benjamin at python.org Mon Jan 11 01:07:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 18:07:35 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Consider this program: class Descr(object): def __init__(self, name): self.name = name def __set__(self, instance, what): instance.__dict__[self.name] = what class X(object): attr = Descr("attr") x = X() print(x.attr) x.attr = 42 print(x.attr) It gives in output: <__main__.Descr object at 0x7fe1c9b28150> 42 The documentation [1] says that Descr is a data descriptor because it defines the __set__ method. It also states that data descriptors always override the value in the instance dictionary. So, the second line should also be the descriptor object according to the documentation. My question is: Is this a doc bug or a implementation bug? If the former, it will be the description of a data descriptor much less consistent, since it will require that a __get__ method be present, too. If the latter, the fix may break some programs relying on the ability to "cache" a value in the instance dictionary. [1] http://docs.python.org/reference/datamodel#invoking-descriptors -- Regards, Benjamin From amauryfa at gmail.com Mon Jan 11 01:51:09 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Jan 2010 01:51:09 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: Hi, 2010/1/11 Benjamin Peterson : > Consider this program: > > class Descr(object): > ? ?def __init__(self, name): > ? ? ? ?self.name = name > ? ?def __set__(self, instance, what): > ? ? ? ?instance.__dict__[self.name] = what > > class X(object): > ? ?attr = Descr("attr") > > x = X() > print(x.attr) > x.attr = 42 > print(x.attr) > > It gives in output: > > <__main__.Descr object at 0x7fe1c9b28150> > 42 > > The documentation [1] says that Descr is a data descriptor because it > defines the __set__ method. It also states that data descriptors > always override the value in the instance dictionary. So, the second > line should also be the descriptor object according to the > documentation. > > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors Quoting the documentation: """Normally, data descriptors define both __get__() and __set__(), while non-data descriptors have just the __get__() method. """ Your example is neither a data descriptor nor a non-data descriptor... The thing that worries me a bit is the "x.attr" returning the Descr object. Descriptors should remain at the class level, and instance should only see values. I'd prefer an AttributeError in this case. -- Amaury Forgeot d'Arc From benjamin at python.org Mon Jan 11 02:00:32 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 10 Jan 2010 19:00:32 -0600 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <1afaf6161001101700u21006708l2ef62bfa257e4ce7@mail.gmail.com> 2010/1/10 Amaury Forgeot d'Arc : > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... See the footnote: http://docs.python.org/reference/datamodel#id7 > > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. Far too late for that, I'm afraid. -- Regards, Benjamin From nas at arctrix.com Mon Jan 11 02:44:57 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 19:44:57 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <20100111014457.GA5289@arctrix.com> On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > I don't think ending the 2.x series at 2.7 makes it look bad > compared to 3.2; it's simply the end of a development line like > any other software project. I suspect 2.7 will have a protracted > bugfix window because so much code runs on 2.x exclusively at the > moment. I would guess over 99% of all Python code written doesn't run on Python 3. Given that, I think it is premature to close the door on new major versions of Python 2.x. Also, we as a project should be careful not to present the image that Python 2.x will not be supported in the future. > If there really is an outcry on this we can re-visit the issue, > but as of right now we need to move forward at some point and 2.7 > seems like that good point. I think that's bad PR. If I had a successful product, I would not announce its end of life just to see how many customers scream and then decide if I should devote more resources to continue maintaining it. IMHO, the release notes should say something like: After the Python 2.7 release, the focus of Python development will be on Python 3. There will continue to be maintainance releases of Python 2.x. trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil From tjreedy at udel.edu Mon Jan 11 03:26:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Jan 2010 21:26:34 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On 1/10/2010 8:44 PM, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: >> I don't think ending the 2.x series at 2.7 makes it look bad >> compared to 3.2; it's simply the end of a development line like >> any other software project. I suspect 2.7 will have a protracted >> bugfix window because so much code runs on 2.x exclusively at the >> moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. If the removal of old features had been done in the 2.x series, as once planned (Guido originally proposed removing the old meaning of int / int in 2.5) the same more or less would be true of 2.7. It is past time for other old and now duplicated features to be removed also. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > >> If there really is an outcry on this we can re-visit the issue, >> but as of right now we need to move forward at some point and 2.7 >> seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. Python is not being ended, but upgraded (with bloating duplications removed). Think of 3.1 as 2.8 with a new name. tjr From lrekucki at gmail.com Mon Jan 11 04:26:48 2010 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Mon, 11 Jan 2010 04:26:48 +0100 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency Message-ID: > Date: Mon, 11 Jan 2010 01:51:09 +0100 > From: "Amaury Forgeot d'Arc" > To: Benjamin Peterson > Cc: Python Dev > Subject: Re: [Python-Dev] Data descriptor doc/implementation > ? ? ? ?inconsistency > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > 2010/1/11 Benjamin Peterson : >> Consider this program: >> >> class Descr(object): >> ? ?def __init__(self, name): >> ? ? ? ?self.name = name >> ? ?def __set__(self, instance, what): >> ? ? ? ?instance.__dict__[self.name] = what >> >> class X(object): >> ? ?attr = Descr("attr") >> >> x = X() >> print(x.attr) >> x.attr = 42 >> print(x.attr) >> >> It gives in output: >> >> <__main__.Descr object at 0x7fe1c9b28150> >> 42 >> >> The documentation [1] says that Descr is a data descriptor because it >> defines the __set__ method. It also states that data descriptors >> always override the value in the instance dictionary. So, the second >> line should also be the descriptor object according to the >> documentation. >> >> My question is: Is this a doc bug or a implementation bug? If the >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors > > Quoting the documentation: > """Normally, data descriptors define both __get__() and __set__(), > while non-data descriptors have just the __get__() method. > """ > Your example is neither a data descriptor nor a non-data descriptor... Actually, there is this footnote [1]: """ A descriptor can define any combination of __get__(), __set__() and __delete__(). If it does not define __get__(), then accessing the attribute even on an instance will return the descriptor object itself. If the descriptor defines __set__() and/or __delete__(), it is a data descriptor; if it defines neither, it is a non-data descriptor. """ Which would mean Descr is actually a data descriptor without a __get__(), so x.attr should always return the descriptor object itself (at least in the docs). > The thing that worries me a bit is the "x.attr" returning the Descr > object. Descriptors should remain at the class level, and instance > should only see values. I'd prefer an AttributeError in this case. > > -- > Amaury Forgeot d'Arc [1] http://docs.python.org/reference/datamodel#id7 -- Lukasz Rekucki From brett at python.org Mon Jan 11 04:46:04 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 19:46:04 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 12:09:08PM -0800, Brett Cannon wrote: > > I don't think ending the 2.x series at 2.7 makes it look bad > > compared to 3.2; it's simply the end of a development line like > > any other software project. I suspect 2.7 will have a protracted > > bugfix window because so much code runs on 2.x exclusively at the > > moment. > > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Well yeah; Python 3.0 is just over a year old with 3.1 -- the first robust 3.x release - is about six months old. From the beginning uptake was expected to take years, not months, so saying that 3.x is not popular enough seems premature. > Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. > No one has said bugfixes will cease. > > > If there really is an outcry on this we can re-visit the issue, > > but as of right now we need to move forward at some point and 2.7 > > seems like that good point. > > I think that's bad PR. If I had a successful product, I would not > announce its end of life just to see how many customers scream and > then decide if I should devote more resources to continue > maintaining it. > > I never said that I wanted to make this announcement specifically to provoke outcries. I said if there happened to be outcries we could possibly visit the issue again. Basically I was being considerate and trying to leave the door open to discuss things in the future even though I don't see the situation changing. > IMHO, the release notes should say something like: > > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. > No because that suggests new features will be coming to 2.x which is not going to happen. If you want to say there will be continual bugfix releases for 2.7 as is par the course for Python and that the number of bugfix releases might be more than normal then I am okay with that. > > > trying-to-head-off-the-python-is-dying-meme-ly y'rs Neil > That came and went already a couple months ago when we discussed stopping at 2.6 instead of 2.7 (see the various threads at http://mail.python.org/pipermail/python-dev/2009-November/thread.html). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 05:05:07 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:05:07 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <20100111040507.GA7200@arctrix.com> On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > After the Python 2.7 release, the focus of Python development > > will be on Python 3. There will continue to be maintainance > > releases of Python 2.x. > > No because that suggests new features will be coming to 2.x which is not > going to happen. If you want to say there will be continual bugfix releases > for 2.7 as is par the course for Python and that the number of bugfix > releases might be more than normal then I am okay with that. Are you are saying that if someone steps up to merge the Unladen Swallow features into a 2.8 release and someone also steps up to cut the release that they will be prevented from doing so? Also, are you presuming to channel the BDFL or was this dicussed somewhere other than python-dev? Maybe I'm overreacting but I get the feeling that the larger and less active segment of the Python develpment team hasn't been involved in these decisions. Best regards, Neil From nas at arctrix.com Mon Jan 11 05:17:54 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 22:17:54 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5C69.7090007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> Message-ID: <20100111041754.GB7200@arctrix.com> On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: > [...] would it still be ok if I closed a 2.x feature request as > "won't fix", or only committed the new feature to the 3.x branch? Yes. Non-bugfix development on 2.x would optional (i.e. done by people who want to spend the time). Since language changes are now out (an initiative I completely support), the majority of non-bugfix changes would be optimizations and platform porting. Regards, Neil From brett at python.org Mon Jan 11 06:06:15 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 21:06:15 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111040507.GA7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 20:05, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 07:46:04PM -0800, Brett Cannon wrote: > > On Sun, Jan 10, 2010 at 17:44, Neil Schemenauer wrote: > > > After the Python 2.7 release, the focus of Python development > > > will be on Python 3. There will continue to be maintainance > > > releases of Python 2.x. > > > > No because that suggests new features will be coming to 2.x which is not > > going to happen. If you want to say there will be continual bugfix > releases > > for 2.7 as is par the course for Python and that the number of bugfix > > releases might be more than normal then I am okay with that. > > Are you are saying that if someone steps up to merge the Unladen > Swallow features into a 2.8 release and someone also steps up to cut > the release that they will be prevented from doing so? > > I honestly don't know, but it's a possibility just like any other new feature request. If people start taking the carrots we have added to 3.x and backporting them to keep the 2.x series alive you are essentially making the 3.x DOA by negating its benefits which I personally don't agree with. > Also, are you presuming to channel the BDFL or was this dicussed > somewhere other than python-dev? This was discussed; see the November 2009 threads. Guido actually suggested ending the 2.x branch at 2.6 until people spoke up about the amount of stuff already backported to 2.7 from 3.x because it was being assumed to be the end of the series to warrant keeping it to help transition to 2.7. > Maybe I'm overreacting but I get > the feeling that the larger and less active segment of the Python > develpment team hasn't been involved in these decisions. > This all came up in November from the 3rd through the 6th (four days) over a ton of email traffic. This was not a snap decision but a heated discussion where even Guido participated that culminated with the decision to stop at 2.7 for the 2.x series; this was not a smoked-filled room decision. I'm sorry if people missed it when they weren't looking, but python-dev is supposed to be the place to watch for this kind of stuff. I don't know how else to bring this stuff to people's attention short of also on python-committers, but developers are basically expected to be on both lists so I don't see where anyone did anything wrong in terms of informing developers. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon Jan 11 06:27:13 2010 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 10 Jan 2010 23:27:13 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> Message-ID: <20100111052713.GA8211@arctrix.com> On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > If people start taking the carrots we have added to 3.x and > backporting them to keep the 2.x series alive you are essentially > making the 3.x DOA by negating its benefits which I personally > don't agree with. I think we have got to the heart of our disagreement. Assume that some superhuman takes all the backwards compatible goodies from 3.x and merges them into 2.x. If that really would kill off Python 3 then I would proclaim it a project that deserved to die. Why should the backwards incompatible changes be endured if they cannot justify their existance by the benefit they provide? I guess I have more confidence in Python 3 than you do. I don't see why Python 2.x needs to be artificially limited so that Python 3 can benefit. Regards, Neil From brett at python.org Mon Jan 11 07:30:49 2010 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2010 22:30:49 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: On Sun, Jan 10, 2010 at 21:27, Neil Schemenauer wrote: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. > > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. If that really would kill off Python 3 > then I would proclaim it a project that deserved to die. Why should > the backwards incompatible changes be endured if they cannot justify > their existance by the benefit they provide? > > When I said "carrots" I meant everything that makes Python 3 the best version of Python, including the backward-incompatible changes. I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. > I don't view it as artificial but simply a focusing of the development team on what has been determined as the future of Python by Guido and letting go of the past. IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying "Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out". That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Jan 11 08:08:14 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 08:08:14 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <319e029f1001102308p5de87cber2131f3aec1abc9f0@mail.gmail.com> On Mon, Jan 11, 2010 at 06:27, Neil Schemenauer wrote: > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. The superhumans of core developers pretty much already have. That's pretty much 2.7 you are talking about. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From chrism at plope.com Mon Jan 11 08:27:03 2010 From: chrism at plope.com (Chris McDonough) Date: Mon, 11 Jan 2010 02:27:03 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4AD2C7.1050703@plope.com> Brett Cannon wrote: > IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev > saying "Python 3 is the future, but we are keeping the old Python 2.x > around because we don't have *that* much faith in the future we have > laid out". That's poison to Python 3 by showing a lack of confidence in > the direction that the BDFL and python-dev as a group has chosen. Now I > could be wrong and there could actually be a large number of active > contributors who want to keep the 2.x series going, but based on the > discussion that occurred the last time this came up I believe the guys > who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort "Python 2". - C From martin at v.loewis.de Mon Jan 11 09:06:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:06:16 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: <4B4ADBF8.3030803@v.loewis.de> Neil Schemenauer wrote: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). I think *that* would give a very bad impression of Python. Depending whom you deal with, the new feature you want may or may not get added to the code base. Contributors would feel even more stranded than they do now, where it may take several years to get a patch reviewed, as you then could submit a patch, and pray that a comitter reviews it who believes in future 2.x releases. The point of setting policies is that it gives every user (contributors, committers, and "end-user" developers) a reliable foundation for expectations. Regards, Martin From arcriley at gmail.com Mon Jan 11 09:06:10 2010 From: arcriley at gmail.com (Arc Riley) Date: Mon, 11 Jan 2010 03:06:10 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AD2C7.1050703@plope.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4AD2C7.1050703@plope.com> Message-ID: after all these years, some people still run AmigaOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 11 09:11:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:11:25 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111014457.GA5289@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> Message-ID: <4B4ADD2D.6070906@v.loewis.de> > I would guess over 99% of all Python code written doesn't run on > Python 3. Given that, I think it is premature to close the door on > new major versions of Python 2.x. Also, we as a project should be > careful not to present the image that Python 2.x will not be > supported in the future. Why that? It is a fact: 2.x will not be supported, in some future. Should we lie to users? > After the Python 2.7 release, the focus of Python development > will be on Python 3. There will continue to be maintainance > releases of Python 2.x. It shouldn't say that, because it wouldn't be true. Regards, Martin From martin at v.loewis.de Mon Jan 11 09:18:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 09:18:30 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <4B4ADED6.5080207@v.loewis.de> > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. Because it takes too much time. Too much of my time, but apparently also too much of other people's time. Of course, the less active fraction of Python contributors may not notice, since they just chose to not contribute (which, of course, is fine). However, asking me to work twice as much as I want to on the project to keep two branches alive is just unfair. This has nothing to do with pushing 3.x, but all with managing available manpower and still providing quality software. Regards, Martin From regebro at gmail.com Mon Jan 11 09:42:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 09:42:51 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> This is what it says now: "Python 2.7 is scheduled to be the last major version in the 2.x series before it moves into 5 years of bugfix-only mode. " And during this discussion it has been noted that others than the core python team might pick up Python 2 and make releases. So maybe we can and this discussion by changing that line in future releases to be: "Python 2.7 is scheduled to be the last major version in the 2.x series released by the Python Software Foundation before it moves into 5 years of bugfix-only mode. " That doesn't exclude *others* making a Python 2.8 that could include all sorts of crazy features. :) This is mainly just to get the pointless discussion over with. The current Python core team don't want to make a 2.8, so that will not happen. Someone else might, but as Chris points out, they won't step up until they have to, which is probably at least two years from now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 10:06:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 10:06:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> Message-ID: <4B4AEA25.7010100@v.loewis.de> > "Python 2.7 is scheduled to be the last major version in the 2.x > series released by the Python Software Foundation before it moves into > 5 years of bugfix-only mode. " > > That doesn't exclude *others* making a Python 2.8 that could include > all sorts of crazy features. :) > > This is mainly just to get the pointless discussion over with. I know you are just looking for a compromise, but this shouldn't be it: the PSF has deliberately stayed out of the actual Python engineering, so the release that Benjamin makes is not done by the PSF (but by Benjamin and his contributors :-). I like the wording as it is (although I find the promise of five years of bug fix releases a bit too strong). Regards, Martin From regebro at gmail.com Mon Jan 11 10:19:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 10:19:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4AEA25.7010100@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> Message-ID: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" wrote: > I know you are just looking for a compromise, but this shouldn't be it: > the PSF has deliberately stayed out of the actual Python engineering, > so the release that Benjamin makes is not done by the PSF (but by > Benjamin and his contributors :-). Hm. Yeah. That's right of course. I started with saying "official", but then I thought "official according to who?". :) So I changed it to mentioning PSF, but that doesn't work either. I guess the current writing as as good as it gets, unless we change "scheduled" to "expected" or something. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From stefan_ml at behnel.de Mon Jan 11 10:42:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:42:05 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111041754.GB7200@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A5C69.7090007@v.loewis.de> <20100111041754.GB7200@arctrix.com> Message-ID: Neil Schemenauer, 11.01.2010 05:17: > On Mon, Jan 11, 2010 at 12:02:01AM +0100, "Martin v. L?wis" wrote: >> [...] would it still be ok if I closed a 2.x feature request as >> "won't fix", or only committed the new feature to the 3.x branch? > > Yes. Non-bugfix development on 2.x would optional (i.e. done by > people who want to spend the time). Note that there's also the time it takes to make a release (usually involving at least one beta release), plus the possibility of introducing bugs while adding new features or even while fixing agreed bugs. All of this takes time in addition to the time people might want to invest in 'real' development on the 2.x trunk. Stefan From stephen at xemacs.org Mon Jan 11 10:59:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Jan 2010 18:59:57 +0900 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111052713.GA8211@arctrix.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> Message-ID: <8763793qsi.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Schemenauer writes: > On Sun, Jan 10, 2010 at 09:06:15PM -0800, Brett Cannon wrote: > > If people start taking the carrots we have added to 3.x and > > backporting them to keep the 2.x series alive you are essentially > > making the 3.x DOA by negating its benefits which I personally > > don't agree with. Well, I think it's *worse* than that, and I don't think you really mean "DOA", anyway. (Feel free to correct me, of course.) The problem I see with backporting lots of stuff, and/or adding new features that aren't in 3.0, to 2.x is that it will make 2.x even cruftier, when it was already crufty enough that Guido (and almost all of python-dev) bit the bullet and said "backward compatibility is no excuse for keeping something in 3.0". That surely means that a lot of python-dev denizens will declare non-support 2.x for x > 7. It's not going to be the gradual migration we've seen over the past few months as active people start to spend more and more time on 3 vs. 2; it will be a watershed. Especially if these are new features merged from outside that the "small active segment" doesn't know anything about. From the users' point of view, that amounts to a *fork*, even if it's internal and "friendly". > I think we have got to the heart of our disagreement. Assume that > some superhuman takes all the backwards compatible goodies from 3.x > and merges them into 2.x. Isn't that a bit ridiculous? I just don't see any evidence that anything like that is going to happen. Worse, if we *assume* it will happen, I don't see any way to assess whether (1) Python 3 goes belly up, (2) there's an effective fork confusing the users and draining the energy of python-dev, or (3) everybody goes "wow!" because it's so cool that everybody wants to keep maintaining an extra 3 branches indefinitely. My opinion is that given the clear direction the "small active segment" is going, telling the users anything but what Brett proposed is disinformation. > I guess I have more confidence in Python 3 than you do. I don't see > why Python 2.x needs to be artificially limited so that Python 3 can > benefit. It's not for Python 3, which you, I, and I'm pretty sure Brett-in-his- heart-of-hearts agree can take care of itself because it is *better* than Python 2. It's for Python, and for the Python community. From walter at livinglogic.de Mon Jan 11 11:37:56 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 11:37:56 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <201001091438.43576.victor.stinner@haypocalc.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> Message-ID: <4B4AFF84.7070206@livinglogic.de> On 09.01.10 14:38, Victor Stinner wrote: > Le samedi 09 janvier 2010 12:18:33, Walter D?rwald a ?crit : >>> Good idea, I choosed open(filename, encoding="BOM"). >> >> On the surface this looks like there's an encoding named "BOM", but >> looking at your patch I found that the check is still done in >> TextIOWrapper. IMHO the best approach would to the implement a *real* >> codec named "BOM" (or "sniff"). This doesn't require *any* changes to >> the IO library. It could even be developed as a standalone project and >> published in the Cheeseshop. > > Why not, this is another solution to the point (2) (Check for a BOM while > reading or detect it before?). Which encoding would be used if there is not > BOM? UTF-8 sounds like a good choice. UTF-8 might be a good choice, are the failback could be specified in the encoding name, i.e. open("file.txt", encoding="BOM-UTF-8") falls back to UTF-8, if there's no BOM at the start. This could be implemented via a custom codec search function (see http://docs.python.org/library/codecs.html#codecs.register for more info). Servus, Walter From regebro at gmail.com Mon Jan 11 12:12:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:12:20 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4AFF84.7070206@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> Message-ID: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: > UTF-8 might be a good choice No, fallback if there is no BOM should be the local settings, just as fallback is today if you don't specify a codec. I mean, what if you want to look for a BOM but fall back to something else? How far will we go with encoding special information in the codecs names? codec='BOM else UTF-16 else locale'? :-) BOM is not a locale, and should not be a locale. Having a locale called BOM is wrong per se. It should either be default to look for a BOM when codec=None, or a special parameter. If none of these are desired, then we need a special function that takes a filename or file handle, and looks for a BOM and returns the codec found or None. But I find that much less natural and obvious than checking the BOM when codec=None. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Mon Jan 11 12:13:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 12:13:00 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <319e029f1001110313u1d839deodfe06a6c7ec423cf@mail.gmail.com> On Mon, Jan 11, 2010 at 12:12, Lennart Regebro wrote: > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. D'oh! I mean codec here obviously. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 13:29:04 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 13:29:04 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4913DF.7050801@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> Message-ID: <4B4B1990.90705@livinglogic.de> On 10.01.10 00:40, "Martin v. L?wis" wrote: >>> How does the requirement that it be implemented as a codec miss the >>> point? >> >> If we want it to be the default, it must be able to fallback on the current >> locale-based algorithm if no BOM is found. I don't think it would be easy for a >> codec to do that. > > Yes - however, Victor currently apparently *doesn't* want it to be the > default, but wants the user to specify encoding="BOM". If so, it isn't > the default, and it is easy to implement as a codec. > >>> FWIW, I agree with Walter that if it is provided through the encoding= >>> argument, it should be a codec. If it is built into the open function >>> (for whatever reason), it must be provided by some other parameter. >> >> Why not simply encoding=None? > > I don't mind. Please re-read Walter's message - it only said that > *if* this is activated through encoding="BOM", *then* it must be > a codec, and could be on PyPI. I don't think Walter was talking about > the case "it is not activated through encoding='BOM'" *at all*. However if this autodetection feature is useful in other cases (no matter how it's activated), it should be a codec, because as part of the open() function it isn't reusable. >> The default value should provide the most useful >> behaviour possible. Forcing users to choose between two different autodetection >> strategies (encoding=None and another one) is a little insane IMO. And encoding="mbcs" is a third option on Windows. > That wouldn't disturb me much. There are a lot of things in that area > that are a little insane, starting with Microsoft Windows :-) ;) Servus, Walter From barry at python.org Mon Jan 11 13:37:46 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 07:37:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4A5DCE.3070909@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: On Jan 10, 2010, at 6:07 PM, Martin v. L?wis wrote: > As for decisions: I don't think there was an official BDFL pronouncent, > but I recall Guido posting a message close to that, proposing that 2.7 > will be a release that will see bug fix releases for an indefinite > period of time (where indefinite != infinite). This was shortly after > him proposing that perhaps we shouldn't make a 2.7 release at all, and > stop at 2.6. IMO, the focus for Python 2.7 (and beyond) must be on helping people transition to Python 3. That should be the reason why a 2.7 is released and, what should dictate whether a 2.8 is necessary. Maybe everything people need (except manpower and round tuits) is already there. I was pleasantly surprised to find that Distribute supports automatically running 2to3 at install time for Python packages. This allowed me to "port" a recent (admittedly small) package to Python 3 by making it "python2.6 -3" clean and adding a couple of attributes to my setup.py[1]. I don't know how widely that feature's known, but I think it's *huge*, and exactly what we need to be doing. The more we can make it easy for people to port their Python 2 code to Python 3, and to support that with one code base, the quicker we'll get to a predominantly Python 3 world. So the question we should be asking is: what's missing from Python 2.7 to help with this transition? If we can't get it into 2.7, then why, and would pushing it back to 2.8 help at all? -Barry [1] modulo a bug in Distribute that caused doctest in separate files to not be used when running under Python 3. From solipsis at pitrou.net Mon Jan 11 13:39:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 13:39:33 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <1263213574.3327.0.camel@localhost> > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. It is reusable as part of io.TextIOWrapper, though. Regards Antoine. From regebro at gmail.com Mon Jan 11 13:45:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 13:45:30 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B1990.90705@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> Message-ID: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: > However if this autodetection feature is useful in other cases (no > matter how it's activated), it should be a codec, because as part of the > open() function it isn't reusable. But an autodetect feature is not a codec. Sure it should be reusable, but making it a codec seems to be a weird hack to me. And how would you reuse it if it was a codec? A reusable autodetect feature would be useable to detect what codec it is. A autodetect codec would not be useful for that, as it would simply just decode. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From walter at livinglogic.de Mon Jan 11 14:21:15 2010 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 11 Jan 2010 14:21:15 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B25CB.5030003@livinglogic.de> On 11.01.10 13:45, Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 13:29, Walter D?rwald wrote: >> However if this autodetection feature is useful in other cases (no >> matter how it's activated), it should be a codec, because as part of the >> open() function it isn't reusable. > > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. I think we already had this discussion two years ago in the context of XML decoding ;): http://mail.python.org/pipermail/python-dev/2007-November/075138.html > And how would > you reuse it if it was a codec? A reusable autodetect feature would be > useable to detect what codec it is. A autodetect codec would not be > useful for that, as it would simply just decode. I have implemented an XML codec (as part of XIST: http://pypi.python.org/pypi/ll-xist), that can do that: >>> from ll import xml_codec >>> import codecs >>> c = codecs.getincrementaldecoder("xml")() >>> c.encoding >>> c.decode(">> c.encoding >>> c.decode(" version='1.0'") u'' >>> c.encoding >>> c.decode(" encoding='iso-8859-1'?>") u"" >>> c.encoding 'iso-8859-1' Servus, Walter From regebro at gmail.com Mon Jan 11 14:47:12 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 14:47:12 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B25CB.5030003@livinglogic.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B25CB.5030003@livinglogic.de> Message-ID: <319e029f1001110547n1d1c9831p34b0eac519d13f9d@mail.gmail.com> On Mon, Jan 11, 2010 at 14:21, Walter D?rwald wrote: > I think we already had this discussion two years ago in the context of > XML decoding ;): Yup. Ans Martins answer then is my answer now: "> So the code is good, if it is inside an XML parser, and it's bad if it > is inside a codec? Exactly so. This functionality just *isn't* a codec - there is no encoding. Instead, it is an algorithm for *detecting* an encoding." The conclusion was that a method do autodetect encodings would be good. I think the same conclusion applies here. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Mon Jan 11 18:16:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:16:37 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <4B46F67F.60604@v.loewis.de> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> Message-ID: <4B4B5CF5.50806@v.loewis.de> > But an autodetect feature is not a codec. Sure it should be reusable, > but making it a codec seems to be a weird hack to me. Well, the existing UTF-16 codec also is an autodetect feature (to detect the endianness), and I don't consider it a weird hack. Regards, Martin From regebro at gmail.com Mon Jan 11 18:27:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:27:01 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B5CF5.50806@v.loewis.de> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> Message-ID: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> On Mon, Jan 11, 2010 at 18:16, "Martin v. L?wis" wrote: >> But an autodetect feature is not a codec. Sure it should be reusable, >> but making it a codec seems to be ?a weird hack to me. > > Well, the existing UTF-16 codec also is an autodetect feature (to > detect the endianness), and I don't consider it a weird hack. So the BOM codec should raise a UnicodeDecodeError if there is no BOM? Because that's what it would have to do, in that case, because it can't fall back on anything, it has to handle and implement all encodings that have a BOM. And is it then actually very useful? You would have to do a try/except first with encoding='BOM' and then encoding=None to get the fallback to the standard. I must say that I find this whole thing pretty obvious. 'BOM' is not an encoding. Either there should be a method to get the encoding from the BOM, returning None of there isn't one, or open() should look at the BOM when you pass in encoding=None. Or both. That covers all usecases, is easy and obvious. Either open(file=foo, encoding=None) or open(file, encoding=encoding_from_bom(file)) I can't see that open(file, encoding='BOM') has any benefit over this, covers any extra usecase and is clearer in any way. Instead it adds something confusing: An encoding that isn't an encoding. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From python at mrabarnett.plus.com Mon Jan 11 18:35:35 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:35:35 +0000 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <201001091438.43576.victor.stinner@haypocalc.com> <4B4AFF84.7070206@livinglogic.de> <319e029f1001110312t461dc126o1dcb5de381684e3@mail.gmail.com> Message-ID: <4B4B6167.40606@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 11:37, Walter D?rwald wrote: >> UTF-8 might be a good choice > > No, fallback if there is no BOM should be the local settings, just as > fallback is today if you don't specify a codec. > I mean, what if you want to look for a BOM but fall back to something > else? How far will we go with encoding special information in the > codecs names? codec='BOM else UTF-16 else locale'? :-) > > BOM is not a locale, and should not be a locale. Having a locale > called BOM is wrong per se. It should either be default to look for a > BOM when codec=None, or a special parameter. If none of these are > desired, then we need a special function that takes a filename or file > handle, and looks for a BOM and returns the codec found or None. But > I find that much less natural and obvious than checking the BOM when codec=None. > Or pass a function that accepts a byte stream or the first few bytes and returns the encoding and any unused bytes (because the byte stream might not be seekable)? def guess_encoding(byte_stream): data = byte_stream.read(2) if data == b"\xFE\xFF": return "UTF-16BE", b"" return "UTF-8", data text_file = open(filename, encoding=guess_encoding) ... From python at mrabarnett.plus.com Mon Jan 11 18:46:30 2010 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 11 Jan 2010 17:46:30 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> Message-ID: <4B4B63F6.1020309@mrabarnett.plus.com> Lennart Regebro wrote: > On Mon, Jan 11, 2010 at 10:06, "Martin v. L?wis" > wrote: >> I know you are just looking for a compromise, but this shouldn't be >> it: the PSF has deliberately stayed out of the actual Python >> engineering, so the release that Benjamin makes is not done by the >> PSF (but by Benjamin and his contributors :-). > > Hm. Yeah. That's right of course. I started with saying "official", > but then I thought "official according to who?". :) "Official" is whatever the BDFL says it is! :-) > So I changed it to mentioning PSF, but that doesn't work either. I > guess the current writing as as good as it gets, unless we change > "scheduled" to "expected" or something. > From martin at v.loewis.de Mon Jan 11 18:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 18:59:08 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <201001081140.28124.victor.stinner@haypocalc.com> <4B486609.2050804@livinglogic.de> <4B48E277.9010408@v.loewis.de> <4B4913DF.7050801@v.loewis.de> <4B4B1990.90705@livinglogic.de> <319e029f1001110445s6d892da3s226443383072a1b0@mail.gmail.com> <4B4B5CF5.50806@v.loewis.de> <319e029f1001110927y36d82b3ftdf5ff56f24e57c4c@mail.gmail.com> Message-ID: <4B4B66EC.7000203@v.loewis.de> > I must say that I find this whole thing pretty obvious. 'BOM' is not > an encoding. That I certainly agree with. > That covers all usecases, is easy and obvious. Either open(file=foo, > encoding=None) or open(file, encoding=encoding_from_bom(file)) > > I can't see that open(file, encoding='BOM') has any benefit over this, Well, it would have the advantage that Walter pointed out: you can implement it independent of the open() implementation, and even provide it in older versions of Python. Regards, Martin From regebro at gmail.com Mon Jan 11 18:59:52 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Jan 2010 18:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B63F6.1020309@mrabarnett.plus.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> Message-ID: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> On Mon, Jan 11, 2010 at 18:46, MRAB wrote: > "Official" is whatever the BDFL says it is! :-) Heh, right. So, it should say "Guido wants 2.7 to be the last main version of Python 2, so it probably will be. We promise to release bugfixes it for, like, ages". No need to be needlessly formal. :-D -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From olemis at gmail.com Mon Jan 11 19:58:01 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 13:58:01 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: References: <201001080110.35800.victor.stinner@haypocalc.com> Message-ID: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting with a BOM, read()/readline() returns also the BOM whereas the >> BOM should be "ignored". >> [...] > I had similar issues too (please read below ;o) ... On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. And for the other two, perhaps it would make more sense to have > a separate encoding-guessing function that takes a binary stream and > returns a text stream wrapping it with the proper encoding? > About guessing the encoding, I experienced this issue while I was developing a Trac plugin. What I was doing is as follows : - I guessed the MIME type + charset encoding using Trac MIME API (it was a CSV file encoded using UTF-16) - I read the file using `open` - Then wrapped the file using `codecs.EncodedFile` - Then used `csv.reader` ... and still get the BOM in the first value of the first row in the CSV file. {{{ #!python >>> mimetype 'utf-16-le' >>> ef = EncodedFile(f, 'utf-8', mimetype) }}} IMO I think I am +1 for leaving `open` just like it is, and use module `codecs` to deal with encodings, but I am strongly -1 for returning the BOM while using `EncodedFile` (mainly because encoding is explicitly supplied in ;o) > --Guido > CMIIW anyway ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mal at egenix.com Mon Jan 11 21:44:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Jan 2010 21:44:34 +0100 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> Message-ID: <4B4B8DB2.1060102@egenix.com> Olemis Lang wrote: >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>> BOM should be "ignored". >>> > [...] >> > > I had similar issues too (please read below ;o) ... > > On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >> talk. And for the other two, perhaps it would make more sense to have >> a separate encoding-guessing function that takes a binary stream and >> returns a text stream wrapping it with the proper encoding? >> > > About guessing the encoding, I experienced this issue while I was > developing a Trac plugin. What I was doing is as follows : > > - I guessed the MIME type + charset encoding using Trac MIME API (it > was a CSV file encoded using UTF-16) > - I read the file using `open` > - Then wrapped the file using `codecs.EncodedFile` > - Then used `csv.reader` > > ... and still get the BOM in the first value of the first row in the CSV file. You didn't say, but I presume that the charset guessing logic returned either 'utf-16-le' or 'utf-16-be' - those encodings don't remove the leading BOM. The 'utf-16' codec will remove the BOM. > {{{ > #!python > >>>> mimetype > 'utf-16-le' >>>> ef = EncodedFile(f, 'utf-8', mimetype) > }}} Same here: the UTF-8 codec will not remove the BOM, you have to use the 'utf-8-sig' codec for that. > IMO I think I am +1 for leaving `open` just like it is, and use module > `codecs` to deal with encodings, but I am strongly -1 for returning > the BOM while using `EncodedFile` (mainly because encoding is > explicitly supplied in ;o) Note that EncodedFile() doesn't do any fancy BOM detection or filtering. This is the job of the codecs. Also note that BOM removal is only valid at the beginning of a file. All subsequent BOM-bytes have to be read as-is (they map to a zero-width non-breaking space) - without removing them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 11 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Jan 11 22:12:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 07:12:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> Message-ID: <4B4B9447.4060101@gmail.com> Benjamin Peterson wrote: > My question is: Is this a doc bug or a implementation bug? If the > former, it will be the description of a data descriptor much less > consistent, since it will require that a __get__ method be present, > too. If the latter, the fix may break some programs relying on the > ability to "cache" a value in the instance dictionary. > > [1] http://docs.python.org/reference/datamodel#invoking-descriptors I would call it a documentation bug: by leaving out the __get__ you're telling Python to override *just* the setting of the attribute, while still using the normal lookup process for retrieving the attribute (i.e. allowing the attribute to be shadowed in the instance dictionary). Adding a "print('Descriptor Invoked')" to the __set__ method in your example: >>> x = X() >>> print(x.attr) <__main__.Descr object at 0x7f283b51ac50> >>> x.attr = 42 Descriptor invoked >>> print(x.attr) 42 >>> x.attr = 6*9 Descriptor invoked >>> print(x.attr) 54 Note that the behaviour here is still different from that of a data descriptor: with a data descriptor, once it gets shadowed in the instance dictionary, the descriptor is ignored *completely*. The only way to get the descriptor involved again is to eliminate the shadowing. The non-data descriptor with only __set__ is just choosing not to override the attribute lookup process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Mon Jan 11 22:29:38 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 11 Jan 2010 16:29:38 -0500 Subject: [Python-Dev] Improve open() to support reading file starting with an unicode BOM In-Reply-To: <4B4B8DB2.1060102@egenix.com> References: <201001080110.35800.victor.stinner@haypocalc.com> <24ea26601001111058g7f9da075m70e765d6ba8b2086@mail.gmail.com> <4B4B8DB2.1060102@egenix.com> Message-ID: <24ea26601001111329i7f609aa1i9f5db7eb61f1fae7@mail.gmail.com> Probably one part of this is OT , but I think it could complement the discussion ;o) On Mon, Jan 11, 2010 at 3:44 PM, M.-A. Lemburg wrote: > Olemis Lang wrote: >>> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >>> wrote: >>>> Hi, >>>> >>>> Builtin open() function is unable to open an UTF-16/32 file starting with a >>>> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >>>> file starting with a BOM, read()/readline() returns also the BOM whereas the >>>> BOM should be "ignored". >>>> >> [...] >>> >> >> I had similar issues too (please read below ;o) ... >> >> On Thu, Jan 7, 2010 at 7:52 PM, Guido van Rossum wrote: >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it would make more sense to have >>> a separate encoding-guessing function that takes a binary stream and >>> returns a text stream wrapping it with the proper encoding? >>> >> >> About guessing the encoding, I experienced this issue while I was >> developing a Trac plugin. What I was doing is as follows : >> >> - I guessed the MIME type + charset encoding using Trac MIME API (it >> was a CSV file encoded using UTF-16) >> - I read the file using `open` >> - Then wrapped the file using `codecs.EncodedFile` >> - Then used `csv.reader` >> >> ... and still get the BOM in the first value of the first row in the CSV file. > > You didn't say, but I presume that the charset guessing logic > returned either 'utf-16-le' or 'utf-16-be' Yes. In fact they return the full mimetype 'text/csv; charset=utf-16-le' ... ;o) > - those encodings don't > remove the leading BOM. ... and they should ? > The 'utf-16' codec will remove the BOM. > In this particular case there's nothing I can do, I have to process whatever charset is detected in the input ;o) >> {{{ >> #!python >> >>>>> mimetype >> 'utf-16-le' >>>>> ef = EncodedFile(f, 'utf-8', mimetype) >> }}} > > Same here: the UTF-8 codec will not remove the BOM, you have > to use the 'utf-8-sig' codec for that. > >> IMO I think I am +1 for leaving `open` just like it is, and use module >> `codecs` to deal with encodings, but I am strongly -1 for returning >> the BOM while using `EncodedFile` (mainly because encoding is >> explicitly supplied in ;o) > > Note that EncodedFile() doesn't do any fancy BOM detection or > filtering. ... directly. > This is the job of the codecs. > OK ... to come back to the scope of the subject, in the general case, I think that BOM (if any) should be handled by codecs, and therefore indirectly by EncodedFile . If that's a explicit way of working with encodings I'd prefer to use that wrapper explicitly in order to (encode | decode) the file and let the codec detect whether there's a BOM or not and ?adjust? `tell`, `read` and everything else in that wrapper (instead of `open`). > Also note that BOM removal is only valid at the beginning of > a file. All subsequent BOM-bytes have to be read as-is (they > map to a zero-width non-breaking space) - without removing them. > ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Test cases for custom query (i.e report 9) ... PASS (1.0.0) - http://simelo.hg.sourceforge.net/hgweb/simelo/trac-gviz/rev/d276011e7297 From martin at v.loewis.de Mon Jan 11 22:42:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:42:45 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> Message-ID: <4B4B9B55.1010709@v.loewis.de> > So the question we should be asking is: what's missing from Python > 2.7 to help with this transition? Wrt. the distribute feature you've noticed: Python 3.0 already supports that in distutils. So from day one of Python 3.x, you could have used that approach for porting Python code. I had been promoting it ever since, but it's taking up only slowly, partially because it has competition in the approaches "burn the bridges" (i.e. convert to 3.x once, and then have two code bases, or abandon 2.x), and "avoid 2to3" (i.e. try to write code that works in both 2.x and 3.x unmodified). > If we can't get it into 2.7, then > why, and would pushing it back to 2.8 help at all? I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: a) telling people that they have to move to 2.6 first actually hurts migration, instead of helping, because it implies to them that they have to drop old versions (e.g. 2.3.) - just because they had *always* dropped old versions before supporting new ones. b) IMO, people also don't gain much by first migrating to 2.6. In principle, it gives them the opportunity to get py3k warnings. However, I haven't heard a single positive report where these warnings have actually helped people in porting. Yours is the first report saying that you followed the official guideline, but you didn't state whether doing so actually helped (or whether you just ported to 2.6 because the guideline told you to). c) whatever 2.7 does (except perhaps for the warnings), it won't be useful to applications, because they couldn't use it, anyway: they need to support 2.4 and 2.5, and it won't have any of the gimmicks people come up with for 2.7. Adding gimmicks to 2.7 might actually hurt porting: people may have to special-case 2.7 because their work-arounds for older versions may break in 2.7 (e.g. testing whether a name is *not* defined, when it becomes defined in 2.7), plus it gives them an incentive to not port yet until they can drop support for anything before 2.7. Inherently, 2.8 can't improve on that. Regards, Martin From martin at v.loewis.de Mon Jan 11 22:44:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 11 Jan 2010 22:44:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> Message-ID: <4B4B9BB8.3070505@v.loewis.de> > So, it should say "Guido wants 2.7 to be the last main version of > Python 2, so it probably will be. We promise to release bugfixes it > for, like, ages". > > No need to be needlessly formal. :-D That summarizes my understanding of what Guido said fairly well :-) +1 for putting it into the announcements. Regards, Martin From fuzzyman at voidspace.org.uk Mon Jan 11 23:11:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 11 Jan 2010 22:11:44 +0000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4B9447.4060101@gmail.com> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> Message-ID: <4B4BA220.20701@voidspace.org.uk> On 11/01/2010 21:12, Nick Coghlan wrote: > Benjamin Peterson wrote: > > My question is: Is this a doc bug or a implementation bug? If the > >> former, it will be the description of a data descriptor much less >> consistent, since it will require that a __get__ method be present, >> too. If the latter, the fix may break some programs relying on the >> ability to "cache" a value in the instance dictionary. >> >> [1] http://docs.python.org/reference/datamodel#invoking-descriptors >> > [snip...] > > Note that the behaviour here is still different from that of a data > descriptor: with a data descriptor, once it gets shadowed in the > instance dictionary, the descriptor is ignored *completely*. The only > way to get the descriptor involved again is to eliminate the shadowing. > The non-data descriptor with only __set__ is just choosing not to > override the attribute lookup process. > > Does that mean we need a third class of descriptors that are neither data descriptors nor non-data descriptors? What should we call them: really-not-data-descriptors? All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Mon Jan 11 23:55:01 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Jan 2010 14:55:01 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9BB8.3070505@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: +1 from me. (With the caveat that "time will tell" is definitely the ultimate answer. Plans may change unexpectedly, even though we don't expect them to.) PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not helpful. But I trust he has ported a lot more code to 3.x than I have personally. Are there other experiences? --Guido On Mon, Jan 11, 2010 at 1:44 PM, "Martin v. L?wis" wrote: >> So, it should say "Guido wants 2.7 to be the last main version of >> Python 2, so it probably will be. We promise to release bugfixes it >> for, like, ages". >> >> No need to be needlessly formal. :-D > > That summarizes my understanding of what Guido said fairly well :-) > > +1 for putting it into the announcements. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Jan 12 00:16:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 00:16:47 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <4B4BB15F.5020807@v.loewis.de> Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. That's really because I haven't even attempted to use it. Instead, the software would fall flat on its own when running the test suite in 3.x. IOW, I don't need 2.6 to tell me what might break when I can see 3.x actually breaking. Regards, Martin From david.lyon at preisshare.net Tue Jan 12 00:22:42 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:22:42 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4ADED6.5080207@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> Message-ID: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Hi Martin, > Of course, the less active fraction of Python contributors may not > notice, since they just chose to not contribute (which, of course, > is fine). However, asking me to work twice as much as I want to > on the project to keep two branches alive is just unfair. Totally true. Actually as an end-developer I'd say that python 2.x series from a programming perspective is quite good. It doesn't need the addition of a lot of new features imho. So for me, I think too much time spent there would be not yield great benefits. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Well, we all know your work is super quality. :-) That's not being contested. However, Quality can be measured different ways and it can be assessed in different ways. Quality itself is a subjective thing. The point I'm only making is that if a piece of software doesn't have "new" things added over time, then users can get a reverse impression of a lack of quality. We've all seen where 'internal' quality can increase and user perceptions can decrease. It could be things like improved graphics and things readily apparent to the user. At the moment, I would say that the "internal" quality of the python 2.x series is super high. "external" quality issues such as the packaging dilemma give the user the opposite quality experience. But people are working on that as best they can elsewhere. I'll leave it at that. > This has nothing to do with pushing 3.x, but all with managing > available manpower and still providing quality software. Python 3.x needs more carrots. >From an ordinary (perphaps ignorant) user perspective there is nothing. Yes, we know if we actually will start programming then we will like it more. But my wishes to Santa Claus would be allow the free flow of PEPs for Python 3 packaging. Even encourage it. As an end developer, here's what I'd like from Santa in 2010 to get me to swap to python 3: * get all the packages on pypi tested for python 3 * put a web based package manager in python 3. This would perhaps be based around PIP (keep many people happy) and would look much like the built in web-console that you get with a $200 router. * Incorporate SCM based end-user software installs by adding to python3-stdlib the SCM support packages (cvs,bzr,svn,hg). This would *really* help in the scientific community. * put a web interface on distutils also so that we don't have to use a command line interface. (I want a pic of a smiley girl to great me when I build something - "Are you ready to build now Honey?"). ok - I joke. But the point is made. So, ok, maybe these things aren't about 'code' quality. But rather user experience. Things like these do count also as "quality" via the technical term "perception of quality". If the PEP process is as unblocked as the documentation implies, implying that anybody can contribute to Python 3. Then there shouldn't be any issue. David From solipsis at pitrou.net Tue Jan 12 00:37:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Jan 2010 23:37:47 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon preisshare.net> writes: > > > This has nothing to do with pushing 3.x, but all with managing > > available manpower and still providing quality software. > > Python 3.x needs more carrots. As someone who experiences the difference almost every day, I can say 3.x definitely has enough carrots for me. The simple fact that I will be able to raise exceptions with non-ASCII error messages without having to workaround a hopelessly broken conversion/reporting logic is a huge improvement. And that's only a small part of the picture. Regards Antoine. From janssen at parc.com Tue Jan 12 00:47:55 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 11 Jan 2010 15:47:55 PST Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <17215.1263253675@parc.com> > David Lyon preisshare.net> writes: > > > > > This has nothing to do with pushing 3.x, but all with managing > > > available manpower and still providing quality software. > > > > Python 3.x needs more carrots. I'd be happy to move UpLib to Python 3, when the various packages that I need are also on Python 3. And that depresses me to think about. I need Medusa ReportLab PyLucene Email Vobject Mutagen Pyglet Hachoir Win32 I'm on the mailing lists for a lot of these, and the only one that I know is thinking about Python 3 is Email. I'd expect Win32 and PIL to also be thinking about it, but I haven't heard anything. Bill From david.lyon at preisshare.net Tue Jan 12 00:47:51 2010 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 12 Jan 2010 10:47:51 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <1220.218.214.45.58.1263253671.squirrel@syd-srv02.ezyreg.com> Antoine writes: > As someone who experiences the difference almost every day, I can say 3.x > definitely has enough carrots for me. The simple fact that I will be able > to > raise exceptions with non-ASCII error messages without having to > workaround a > hopelessly broken conversion/reporting logic is a huge improvement. And > that's > only a small part of the picture. In no way could I disagree with you. Ascii works fine for us here - but I guess we're lucky. Just keep pushing python 3 and have a nice day.. David From barry at python.org Tue Jan 12 01:11:28 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:11:28 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100111191128.39020d89@freewill> On Jan 11, 2010, at 10:42 PM, Martin v. L?wis wrote: >Wrt. the distribute feature you've noticed: Python 3.0 already supports >that in distutils. So from day one of Python 3.x, you could have used >that approach for porting Python code. I had been promoting it ever >since, but it's taking up only slowly, partially because it has >competition in the approaches "burn the bridges" (i.e. convert to 3.x >once, and then have two code bases, or abandon 2.x), and "avoid 2to3" >(i.e. try to write code that works in both 2.x and 3.x unmodified). Interesting. The only reason I never used it before is because I didn't know about it. Am I alone? Maybe I'm projecting my own preferences, but it seems to me that supporting both Python 2 and 3 from the same code base would be a very powerful way to enable a large amount of existing code to support Python 3. I'm certainly going to do this from now on with all the libraries I maintain. I don't have any cycles to write code twice and I can't abandon Python 2 yet. I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. As an example, the one library I've already ported used a metaclass. I don't see any way to specify that the metaclass should be used in a portable way. In Python 2.6 it's: class Foo: __metaclass__ = Meta and in Python 3 it's: class Foo(metaclass=Meta): 2to3 made that pain go away. >I've done a fair bit of 3.x porting, and I'm firmly convinced that >2.x can do nothing: > >a) telling people that they have to move to 2.6 first actually > hurts migration, instead of helping, because it implies to them > that they have to drop old versions (e.g. 2.3.) - just because > they had *always* dropped old versions before supporting new ones. Is it just an implication, or is it reality? I have yet to try to support older versions of Python and also support Python 3. (The one package I've done this with so far only supports Python 2.6.) >b) IMO, people also don't gain much by first migrating to 2.6. > In principle, it gives them the opportunity to get py3k warnings. > However, I haven't heard a single positive report where these > warnings have actually helped people in porting. Yours is the > first report saying that you followed the official guideline, > but you didn't state whether doing so actually helped (or whether > you just ported to 2.6 because the guideline told you to). Python 2.6 has other useful features, which I want to take advantage of, so getting py3k warnings is a bonus. Running 'python2.6 -3 setup.py test' over my code did in fact help clean up a couple of problems. Seems like a good first step when considering Python 3 support to me. >c) whatever 2.7 does (except perhaps for the warnings), it won't > be useful to applications, because they couldn't use it, anyway: > they need to support 2.4 and 2.5, and it won't have any of the > gimmicks people come up with for 2.7. Adding gimmicks to 2.7 > might actually hurt porting: people may have to special-case > 2.7 because their work-arounds for older versions may break in 2.7 > (e.g. testing whether a name is *not* defined, when it becomes > defined in 2.7), plus it gives them an incentive to not port > yet until they can drop support for anything before 2.7. > >Inherently, 2.8 can't improve on that. I'm not so sure. Yes, as a package maintainer there are older versions to think about, but time moves on for everyone (hopefully :). By the time 2.8 is released, what will be the minimum version of Python provided by most OS vendors (where the majority of Python users probably get their 'python')? I guess some people will have to support everything from Python 2.3 to 2.8 but you're talking supporting something like a spread of 7 years of Python versions. What other platform do you support for 7 years? Even Ubuntu long term support (LTS) releases only live for 5 years and even then, newer Pythons are often back ported. Or if not, then who cares? You're not going to use a newer version of a library on an LTS anyway. I'm not necessarily saying that justifies a 2.8 release. I'm just asking how we can make it easier for people to port to Python 3. The automatic 2to3 has already made it easier for me to port one library to Python 3 because I barely had to even think about it. Maybe that's enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 01:12:19 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Jan 2010 19:12:19 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <20100111191219.5fdd2542@freewill> On Jan 11, 2010, at 02:55 PM, Guido van Rossum wrote: >PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not >helpful. But I trust he has ported a lot more code to 3.x than I have >personally. Are there other experiences? I've only done one small library so far, but it was helpful to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From andrew at bemusement.org Tue Jan 12 01:07:26 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Tue, 12 Jan 2010 11:07:26 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4B9B55.1010709@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> Message-ID: <20100112000726.GO32351@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > I've done a fair bit of 3.x porting, and I'm firmly convinced that > 2.x can do nothing: [...] > Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. From solipsis at pitrou.net Tue Jan 12 01:28:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 00:28:26 +0000 (UTC) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: Andrew Bennetts bemusement.org> writes: > > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. I don't think a 2.8 could make you much closer to 3.x. AFAICT, every possible way to ease transition to 3.x will already be in 2.7, or almost. But perhaps I'm lacking imagination. Regards Antoine. From brian.curtin at gmail.com Tue Jan 12 02:57:46 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 11 Jan 2010 19:57:46 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > -Brett > I think it would be interesting to see how people are using the tracker, or how they want to be using it. For example, there are currently over 1500 open issues with no stage set, some of which seemingly haven't been read by anyone at all. Would a properly set stage field save issues from falling into a black hole? Food for thought: according to the last tracker summary, there are over 1000 open issues with a patch, and issues stay open an average of 700 days (median 450). Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Jan 12 04:53:24 2010 From: jackdied at gmail.com (Jack Diederich) Date: Mon, 11 Jan 2010 22:53:24 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. [sidebar] 1) the metaclass fixer was a PITA to implement. 2) 95% of __metaclass__ definitions searchable via google code were of the "__metaclass__ = type" variety. The 2to3 patch exists only because of the few other uses. 3) 100% of the module level assignments in public projects were the "__metaclass__ = type" variety which is why there isn't a fixer for that. Also, a fixer would have been really, really ugly (munge every class definition in this module because there is a top level assignment). -Jack From benjamin at python.org Tue Jan 12 05:01:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 11 Jan 2010 22:01:40 -0600 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <1afaf6161001112001t5e7bab83t7d93f33fa11ce849@mail.gmail.com> 2010/1/11 Jack Diederich : > On Mon, Jan 11, 2010 at 7:11 PM, Barry Warsaw wrote: >> As an example, the one library I've already ported used a metaclass. ?I don't >> see any way to specify that the metaclass should be used in a portable way. >> In Python 2.6 it's: >> >> class Foo: >> ? ?__metaclass__ = Meta >> >> and in Python 3 it's: >> >> class Foo(metaclass=Meta): >> >> 2to3 made that pain go away. > > [sidebar] > 1) the metaclass fixer was a PITA to implement. Does this make it any less useful, though? :) -- Regards, Benjamin From steven.bethard at gmail.com Tue Jan 12 06:57:18 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 11 Jan 2010 21:57:18 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: On Mon, Jan 11, 2010 at 4:11 PM, Barry Warsaw wrote: > As an example, the one library I've already ported used a metaclass. ?I don't > see any way to specify that the metaclass should be used in a portable way. > In Python 2.6 it's: > > class Foo: > ? ?__metaclass__ = Meta > > and in Python 3 it's: > > class Foo(metaclass=Meta): > > 2to3 made that pain go away. Actually there's a solution to this one too: FooBase = Meta('FooBase', (), {}) class Foo(FooBase): ... That should work in Python 2.X and 3.X. I've got argparse running on Python 2.3-3.1, and the changes were pretty easy. You can see them all in the revision here: http://code.google.com/p/argparse/source/detail?r=12 I have aspirations of putting all of the tricks I learned up up on the Wiki somewhere, but I just haven't had the time. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From regebro at gmail.com Tue Jan 12 07:30:10 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:30:10 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> Message-ID: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> On Mon, Jan 11, 2010 at 23:55, Guido van Rossum wrote: > +1 from me. (With the caveat that "time will tell" is definitely the > ultimate answer. Plans may change unexpectedly, even though we don't > expect them to.) > > PS. I'm surprised that Martin thinks that the -3 mode in 2.6 is not > helpful. But I trust he has ported a lot more code to 3.x than I have > personally. Are there other experiences? It doesn't warn for that many of the unportable problems, but I'm not sure it can warn for them either. It's certainly helpful, just not very much. :) I think the biggest help was added in 2.6.2, and that's warning for old integer division. It will also warn for modules that aren't supported anymore, if I remember correctly, and that's often helpful. But it won't warn for real tricky problems, like binary vs text in strings, and I don't see how it can either. And, I just realized, it doesn't warn for you using cmp or __cmp__ either, and 2to3 won't fix that, so it should actually warn for it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:49:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:49:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> On Tue, Jan 12, 2010 at 01:11, Barry Warsaw wrote: > Interesting. ?The only reason I never used it before is because I didn't know > about it. ?Am I alone? Maybe not, but the Distribute feature is there because IMO the distutils feature by itself isn't particularily useful. You need to write your own distutils extensions, in practice, and they are not trivial. Distribute has simply done it for you. :) > I'm skeptical that code can work unmodified in both 2 and 3 without 2to3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Jan 12 07:53:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 07:53:20 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <319e029f1001112253x511f8109ja2300a022010ef99@mail.gmail.com> On Tue, Jan 12, 2010 at 01:07, Andrew Bennetts wrote: > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. ?I > think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. When 2.7 was discussed several people agreed that 2.7 should include as much 3.x stuff as possible to ease transition. That turned out to not be very much, so I'm not sure there is more. :) Unless of course, 2.8 starts including more of the refactored libraries, but that's a very minor issue in porting, so it won't help much. To really help, it needs to start implement things that break backwards compatibility. That would be weird. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Tue Jan 12 10:39:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:39:39 +1000 Subject: [Python-Dev] Data descriptor doc/implementation inconsistency In-Reply-To: <4B4BA220.20701@voidspace.org.uk> References: <1afaf6161001101607l19860878k7edc431510e2caba@mail.gmail.com> <4B4B9447.4060101@gmail.com> <4B4BA220.20701@voidspace.org.uk> Message-ID: <4B4C435B.7080903@gmail.com> Michael Foord wrote: >> Note that the behaviour here is still different from that of a data >> descriptor: with a data descriptor, once it gets shadowed in the >> instance dictionary, the descriptor is ignored *completely*. The only >> way to get the descriptor involved again is to eliminate the shadowing. >> The non-data descriptor with only __set__ is just choosing not to >> override the attribute lookup process. >> >> > > Does that mean we need a third class of descriptors that are neither > data descriptors nor non-data descriptors? Not really - leaving out __get__ when defining __set__ just creates a data descriptor that happens to use the default attribute look-up process rather than defining a different one. (Note that I had the data/non-data terminology backwards in my previous message - I tend to think of the two kinds of descriptor as enforced and non-enforced respectively precisely because I have to think about it to remember that "functions are non-data descriptors and properties are data descriptors, hence non-data descriptors only define __get__ while data descriptors define __set__". I don't find the data/non-data terminology intuitive at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:44:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:44:18 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <319e029f1001110042ybc57c62w25e380707cd3ae48@mail.gmail.com> <4B4AEA25.7010100@v.loewis.de> <319e029f1001110119x39695088yf85c6ec64b6eba1e@mail.gmail.com> <4B4B63F6.1020309@mrabarnett.plus.com> <319e029f1001110959g68a9f1d9xe40e51f88d6db244@mail.gmail.com> <4B4B9BB8.3070505@v.loewis.de> <319e029f1001112230i3251a1c1k2ab7d159ce755f75@mail.gmail.com> Message-ID: <4B4C4472.10104@gmail.com> Lennart Regebro wrote: > And, I just realized, it doesn't warn for you using cmp or __cmp__ > either, and 2to3 won't fix that, so it should actually warn for it. I have a vague recollection that we tried to warn for that and ended up nixing the warning due to vast swarms of false alarms (or because you couldn't write correct 2.x code without triggering it). I'd be happy for someone to prove my recollection wrong though (i.e. by writing a patch that implements such warnings). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jan 12 10:48:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 19:48:57 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B4C4589.70906@gmail.com> David Lyon wrote: >> This has nothing to do with pushing 3.x, but all with managing >> available manpower and still providing quality software. > > Python 3.x needs more carrots. As Guido has said a few times, the gains are far greater for *new* Python developers than they are for existing ones. Existing developers have to unlearn old habits and wait for libraries they already use to get ported. New developers can just get started with a much cleaner language. They don't have as rich a 3rd party ecosystem on Python 3 as they would on Python 2.x at this point in time, but unlike existing developers they also have the luxury of cherry-picking just those packages that already have Python 3 support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Tue Jan 12 11:20:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:20:27 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit?: > > For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. I think most issues /have/ been read. It's just that for many of them, nobody is interested enough in or feels competent enough for fixing them. > Would a properly set stage field save issues from > falling into a black hole? I don't think this has anything to do with properly setting the stage field. We just have limited time and manpower. Perhaps one of our goals should be to reach out more to potential contributors. > Food for thought: according to the last tracker summary, there are over > 1000 open issues with a patch, and issues stay open an average of 700 > days (median 450). As for open issues with a patch, a "code review" button sending this patch to Rietveld (or any other similar tool) is something many of us have been hoping for :-) It would make reviewing easier, faster and more thorough (because you visualize things better than by looking at a diff). Regards Antoine. From solipsis at pitrou.net Tue Jan 12 11:36:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Jan 2010 10:36:49 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Le Tue, 12 Jan 2010 10:20:27 +0000, Antoine Pitrou a ?crit?: > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Speaking of which, Steve had something to say about it: http://holdenweb.blogspot.com/2009/11/comments-or-not-public-or- private.html cheers Antoine. From ncoghlan at gmail.com Tue Jan 12 13:10:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Jan 2010 22:10:14 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C66A6.2040601@gmail.com> Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : >> For example, there are currently over >> 1500 open issues with no stage set, some of which seemingly haven't been >> read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. There are actually a whole host of reasons issues can stagnate: - a feature request may seem reasonable (hence it doesn't get rejected outright), but the right API may not be clear (hence it doesn't get implemented in the near term) - a patch may be reviewed and found to have significant defects (or simply be overreaching the stated goal) but the original patch poster loses interest after meeting resistance in their ambition to fix something that is "obviously" broken - a problem may simply be hard to fix in a backwards compatible way (or even at all!) Ultimately it boils down to Antoine's two categories (lack of interest and lack of relevant expertise to make a final decision) but there are a lot of different ways to get into those two buckets. Because we aren't ruthless about pruning those kinds of issues out of the tracker they're the ones that are going to accumulate over time. I'd actually be interested to know what the issue stats are like when RFEs are excluded and when the search is limited to the items flagged as 'easy'. If easy bug fix issues are taking a long time to get closed than that would bother me a lot more than RFEs that sit on the tracker for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 12 13:14:47 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:14:47 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071447.675c8f24@freewill> On Jan 11, 2010, at 10:53 PM, Jack Diederich wrote: >3) 100% of the module level assignments in public projects were the >"__metaclass__ = type" variety which is why there isn't a fixer for >that. Also, a fixer would have been really, really ugly (munge every >class definition in this module because there is a top level >assignment). And almost certainly unnecessary. IME, those are all to easily make bare class definitions new style in Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From barry at python.org Tue Jan 12 13:16:09 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Jan 2010 07:16:09 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <20100112071609.1dcfffa6@freewill> On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: >Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > >That should work in Python 2.X and 3.X. Ugly, but good call! :) >I've got argparse running on Python 2.3-3.1, and the changes were >pretty easy. You can see them all in the revision here: > > http://code.google.com/p/argparse/source/detail?r=12 > >I have aspirations of putting all of the tricks I learned up up on the >Wiki somewhere, but I just haven't had the time. The more resources we can provide people, both in code and in documentation, the better. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Tue Jan 12 13:29:12 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 12:29:12 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112071609.1dcfffa6@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> Message-ID: <4B4C6B18.7050008@voidspace.org.uk> On 12/01/2010 12:16, Barry Warsaw wrote: > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > >> Actually there's a solution to this one too: >> >> FooBase = Meta('FooBase', (), {}) >> class Foo(FooBase): >> ... >> >> That should work in Python 2.X and 3.X. >> > Ugly, but good call! :) > > There are all sorts of tricks. For example you can do exception handling that works with pre-2.6 syntax and 3.0 with a bare except and using sys.exc_info. It is horrible, but acceptable for short pieces of code (I have a couple of small modules that do this). I haven't yet tried converting larger code-bases to Python 3, but I think the workflow advocated by Martin is greatly preferable to the hacks and tricks needed to make the same codebase run under 2 & 3. Michael >> I've got argparse running on Python 2.3-3.1, and the changes were >> pretty easy. You can see them all in the revision here: >> >> http://code.google.com/p/argparse/source/detail?r=12 >> >> I have aspirations of putting all of the tricks I learned up up on the >> Wiki somewhere, but I just haven't had the time. >> > The more resources we can provide people, both in code and in documentation, > the better. > > Thanks! > -Barry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 12 14:09:50 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Jan 2010 14:09:50 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4C749E.4040009@egenix.com> Brett Cannon wrote: > Michael has given me the hg transition/stdlib time slot at the language > summit this year. In regards to that I plan to lead a discussion on: > > * where we are at w/ the Hg transition (Dirkjan should be there and I did a > blog post on this topic recently: > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > * argparse (PEP 389) > * brief mention on still wanting to break out the stdlib from CPython > * an official policy on extension modules? (i.e. must have a pure Python > implementation which can mean a ctypes implementation unless given an > explicit waiver) > > That's everything from a stdlib perspective. I have half-baked ideas, but > nothing concrete enough to bring up unless people really want to discuss > stuff like how to potentially standardize module deprecation warnings, etc. > > In terms of me personally, I do plan to bring up at some point during the > summit these points which don't squarely fit during my time slot: > > * an official unofficial policy on how new proposed features should come to > us (i.e. working code to python-ideas with a shell of a PEP that includes > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > * any changes needed to the issue tracker to help with the workflow? (stage > field seems like a failed experiment and we now have several effective > triage people who can help w/ guiding changes) > > If there is something missing from the stdlib discussion that you think > should be brought up at the summit let me know. And if there is something > here you want to discuss before the summit let me know and I can start a > separate thread on it. Could you please put these things and the results up on the Python wiki ?! We're going to have a language summit at EuroPython this year as well and may want to continue/extend the discussion based on what you're doing at PyCon. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 12 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brian.curtin at gmail.com Tue Jan 12 15:33:06 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 12 Jan 2010 08:33:06 -0600 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Tue, Jan 12, 2010 at 04:20, Antoine Pitrou wrote: > Le Mon, 11 Jan 2010 19:57:46 -0600, Brian Curtin a ?crit : > > > > For example, there are currently over > > 1500 open issues with no stage set, some of which seemingly haven't been > > read by anyone at all. > > I think most issues /have/ been read. It's just that for many of them, > nobody is interested enough in or feels competent enough for fixing them. > > > Would a properly set stage field save issues from > > falling into a black hole? > > I don't think this has anything to do with properly setting the stage > field. We just have limited time and manpower. Perhaps one of our goals > should be to reach out more to potential contributors. Agreed, I didn't mean to place blame on the stage field, I just ran with how I view that field since it was mentioned. When I'm thinking in "code-writing developer mode" (tm), I'm more likely to go to issues which have a stage so I know what I'm going into -- what needs to be worked on. When I'm in project cleanup mode, I go by stageless issues -- what is necessary for this to begin or end work. Maybe others work differently...software projects take all kinds :-) Anyways, sorry for the off-topic. If this is a summit worthy discussion, cool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 17:12:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 11:12:28 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C66A6.2040601@gmail.com> References: <4B4C66A6.2040601@gmail.com> Message-ID: <20100112161228.300EA1D688D@kimball.webabinitio.net> On Tue, 12 Jan 2010 22:10:14 +1000, Nick Coghlan wrote: > There are actually a whole host of reasons issues can stagnate: > - a feature request may seem reasonable (hence it doesn't get rejected > outright), but the right API may not be clear (hence it doesn't get > implemented in the near term) > - a patch may be reviewed and found to have significant defects (or > simply be overreaching the stated goal) but the original patch poster > loses interest after meeting resistance in their ambition to fix > something that is "obviously" broken > - a problem may simply be hard to fix in a backwards compatible way (or > even at all!) I would add: - a patch is reviewed but the reviewer requests a second opinion before committing, which never arrives, and the original reviewer forgets about the issue. I have occasionally observed apparently moribund issues spring to life and get resolved when a triage person does something as simple as updating the releases to which an issue applies. > Ultimately it boils down to Antoine's two categories (lack of interest > and lack of relevant expertise to make a final decision) but there are a > lot of different ways to get into those two buckets. There's a third bucket here: lack of time. Sometimes there is interest and expertise, but no time. Worse, sometimes we end up waiting on the person with the expertise and interest but no time, when someone else with somewhat less expertise but more time should just go ahead and do the commit. (*That* one should be fixable.) > Because we aren't ruthless about pruning those kinds of issues out of > the tracker they're the ones that are going to accumulate over time. Another other category that hangs around in the tracker is items for which there simply isn't a consensus. I suppose those could be brought to python-dev for a final decision if someone wanted to tackle that task. And I don't think it is a lack of ruthlessness. I think it is the current community consensus that we leave these issues open in the tracker in case someone does come along and want to deal with them. (*) > I'd actually be interested to know what the issue stats are like when > RFEs are excluded and when the search is limited to the items flagged as > 'easy'. If easy bug fix issues are taking a long time to get closed than > that would bother me a lot more than RFEs that sit on the tracker for > years. I'd also be interested to know what the average and median lifetime of closed bug reports is. Not the metrics for open issues, but the metrics for closed ones. My theory is that we close a lot of bugs fairly promptly, but the ones in the above categories make the average age of *open* issues high. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls (*) For example, I'm currently going through all the open issues relating to the email package, and some of them are pretty darn old, yet still have useful content. From brett at python.org Tue Jan 12 18:40:13 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:40:13 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <4B4C749E.4040009@egenix.com> References: <4B4C749E.4040009@egenix.com> Message-ID: On Tue, Jan 12, 2010 at 05:09, M.-A. Lemburg wrote: > Brett Cannon wrote: > > Michael has given me the hg transition/stdlib time slot at the language > > summit this year. In regards to that I plan to lead a discussion on: > > > > * where we are at w/ the Hg transition (Dirkjan should be there and I did > a > > blog post on this topic recently: > > http://sayspy.blogspot.com/2010/01/where-hg-transition-stands.html) > > * argparse (PEP 389) > > * brief mention on still wanting to break out the stdlib from CPython > > * an official policy on extension modules? (i.e. must have a pure Python > > implementation which can mean a ctypes implementation unless given an > > explicit waiver) > > > > That's everything from a stdlib perspective. I have half-baked ideas, but > > nothing concrete enough to bring up unless people really want to discuss > > stuff like how to potentially standardize module deprecation warnings, > etc. > > > > In terms of me personally, I do plan to bring up at some point during the > > summit these points which don't squarely fit during my time slot: > > > > * an official unofficial policy on how new proposed features should come > to > > us (i.e. working code to python-ideas with a shell of a PEP that includes > > abstract and motivation -> hashed out PEP to python-dev -> pronouncement) > > * any changes needed to the issue tracker to help with the workflow? > (stage > > field seems like a failed experiment and we now have several effective > > triage people who can help w/ guiding changes) > > > > If there is something missing from the stdlib discussion that you think > > should be brought up at the summit let me know. And if there is something > > here you want to discuss before the summit let me know and I can start a > > separate thread on it. > > Could you please put these things and the results up on the Python > wiki ?! > > We're going to have a language summit at EuroPython this year > as well and may want to continue/extend the discussion based > on what you're doing at PyCon. > > I expect there will be at least summary emails on what gets discussed. There is also a chance that it will be videotaped. -Brett > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 12 2010) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 18:47:50 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 09:47:50 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? >> (stage field seems like a failed experiment and we now have several >> effective triage people who can help w/ guiding changes) >> >> -Brett >> > I think it would be interesting to see how people are using the tracker, or > how they want to be using it. For example, there are currently over 1500 > open issues with no stage set, some of which seemingly haven't been read by > anyone at all. Would a properly set stage field save issues from falling > into a black hole? > > "Using" is the key word there. I know this thread went on about how bugs tend to end up being left open, but what I was proposing to talk about was whether there is any shift desired by the seasoned tracker users to help in their work. For instance, the Stage field was meant to help, but I don't think it is really getting used much, which makes it at best just a UI junk and at worst something to confuse new users. So I just wanted to discuss things as a group to see if we could streamline some things, add others, etc. At worst I will try to grab people like Mark D., R. David, Antoine, Georg, etc. at the summit (not sure exactly which the heavy bug closers will be there), take them aside, and flat-out ask what they need to make their jobs easier. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 12 19:29:06 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:29:06 -0800 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C6B18.7050008@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <20100112071609.1dcfffa6@freewill> <4B4C6B18.7050008@voidspace.org.uk> Message-ID: On Tue, Jan 12, 2010 at 04:29, Michael Foord wrote: > On 12/01/2010 12:16, Barry Warsaw wrote: > > On Jan 11, 2010, at 09:57 PM, Steven Bethard wrote: > > > > Actually there's a solution to this one too: > > FooBase = Meta('FooBase', (), {}) > class Foo(FooBase): > ... > > That should work in Python 2.X and 3.X. > > > Ugly, but good call! :) > > > > > There are all sorts of tricks. For example you can do exception handling > that works with pre-2.6 syntax and 3.0 with a bare except and using > sys.exc_info. It is horrible, but acceptable for short pieces of code (I > have a couple of small modules that do this). > > I haven't yet tried converting larger code-bases to Python 3, but I think > the workflow advocated by Martin is greatly preferable to the hacks and > tricks needed to make the same codebase run under 2 & 3. > > In other words we need to pull together a HOWTO for Python source like the one for extension modules that Benjamin wrote and make it rather prominently linked from the Python 3 documentation index page. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 12 19:31:23 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Jan 2010 13:31:23 -0500 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > On Mon, Jan 11, 2010 at 17:57, Brian Curtin wrote: > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > >> * any changes needed to the issue tracker to help with the workflow? > >> (stage field seems like a failed experiment and we now have several > >> effective triage people who can help w/ guiding changes) > >> > > I think it would be interesting to see how people are using the tracker, or > > how they want to be using it. For example, there are currently over 1500 > > open issues with no stage set, some of which seemingly haven't been read by > > anyone at all. Would a properly set stage field save issues from falling > > into a black hole? > > > "Using" is the key word there. I know this thread went on about how bugs > tend to end up being left open, but what I was proposing to talk about was > whether there is any shift desired by the seasoned tracker users to help in > their work. For instance, the Stage field was meant to help, but I don't > think it is really getting used much, which makes it at best just a UI junk > and at worst something to confuse new users. So I just wanted to discuss > things as a group to see if we could streamline some things, add others, > etc. Well, I for one find the stage field useful. It reminds me at a glance where the issue is at in the workflow (and 'not set' is a valid place in the workflow that an issue sometimes stays in for a while for reason other than not having been triaged yet). I suspect that if we discussed it as a group (we who make the most changes on the tracker) we could improve it, and the interface in general. By the way, you could talk to people who aren't going to be at the summit on #python-dev; I think all the currently tracker-active people hang out there on a regular basis. I'll have to give some thought to what changes/improvements might be most useful, now that I've been doing this for a while. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From brett at python.org Tue Jan 12 19:34:02 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 10:34:02 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> References: <20100112183123.71F4D1FBE4D@kimball.webabinitio.net> Message-ID: On Tue, Jan 12, 2010 at 10:31, R. David Murray wrote: > On Tue, 12 Jan 2010 09:47:50 -0800, Brett Cannon wrote: > > On Mon, Jan 11, 2010 at 17:57, Brian Curtin > wrote: > > > On Sun, Jan 10, 2010 at 14:25, Brett Cannon wrote: > > >> * any changes needed to the issue tracker to help with the workflow? > > >> (stage field seems like a failed experiment and we now have several > > >> effective triage people who can help w/ guiding changes) > > >> > > > I think it would be interesting to see how people are using the > tracker, or > > > how they want to be using it. For example, there are currently over > 1500 > > > open issues with no stage set, some of which seemingly haven't been > read by > > > anyone at all. Would a properly set stage field save issues from > falling > > > into a black hole? > > > > > "Using" is the key word there. I know this thread went on about how bugs > > tend to end up being left open, but what I was proposing to talk about > was > > whether there is any shift desired by the seasoned tracker users to help > in > > their work. For instance, the Stage field was meant to help, but I don't > > think it is really getting used much, which makes it at best just a UI > junk > > and at worst something to confuse new users. So I just wanted to discuss > > things as a group to see if we could streamline some things, add others, > > etc. > > Well, I for one find the stage field useful. It reminds me at a glance > where the issue is at in the workflow (and 'not set' is a valid place > in the workflow that an issue sometimes stays in for a while for reason > other than not having been triaged yet). I suspect that if we discussed > it as a group (we who make the most changes on the tracker) we could > improve it, and the interface in general. > > By the way, you could talk to people who aren't going to be at the > summit on #python-dev; I think all the currently tracker-active people > hang out there on a regular basis. > > Sounds reasonable. > I'll have to give some thought to what changes/improvements might be > most useful, now that I've been doing this for a while. How about everyone who is active on bugs.python.org give it some thought and we can try to have a discussion at some point in the relatively near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jan 12 21:58:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 06:58:49 +1000 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: <4B4C749E.4040009@egenix.com> Message-ID: <4B4CE289.6040709@gmail.com> Brett Cannon wrote: > I expect there will be at least summary emails on what gets discussed. > There is also a chance that it will be videotaped. The Wiki makes for a better summary archive though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jan 12 22:53:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:53:01 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100111191128.39020d89@freewill> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> Message-ID: <4B4CEF3D.8000107@v.loewis.de> >> a) telling people that they have to move to 2.6 first actually >> hurts migration, instead of helping, because it implies to them >> that they have to drop old versions (e.g. 2.3.) - just because >> they had *always* dropped old versions before supporting new ones. > > Is it just an implication, or is it reality? That's only the implication. However, this was precisely the dialogue when talking to Django. If you start with "start supporting 2.6", the immediate response, without listening further, was, "ok, wait until we drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). Then explain it to the individual you are talking to, wait for the next developer of the project step along, and see how he brings up the very same line of thinking (supporting new versions == dropping support for old versions). I think only part of that comes from the maintenance burden. The other part is that they *want* to drop support for old versions, so that they can eventually start using new features (e.g. generator expressions). So they welcome the requirement to support new versions as an excuse to drop old ones ("it is obvious that you have to drop 2.3 to support 3.2"). However, their users then won't let them drop old versions. > >> b) IMO, people also don't gain much by first migrating to 2.6. >> In principle, it gives them the opportunity to get py3k warnings. >> However, I haven't heard a single positive report where these >> warnings have actually helped people in porting. Yours is the >> first report saying that you followed the official guideline, >> but you didn't state whether doing so actually helped (or whether >> you just ported to 2.6 because the guideline told you to). > > Python 2.6 has other useful features, which I want to take advantage of I think you are a minority with that, being able to actually use the 2.6 features already. Many projects can't, as they have to support at least 2.4 still (so the with statement is right out). >> Inherently, 2.8 can't improve on that. > > I'm not so sure. Yes, as a package maintainer there are older versions to > think about, but time moves on for everyone (hopefully :). By the time 2.8 is > released, what will be the minimum version of Python provided by most OS > vendors (where the majority of Python users probably get their 'python')? "Current" Linux distributions will have 2.6 then. "Old" installations will have 2.4. > I > guess some people will have to support everything from Python 2.3 to 2.8 but > you're talking supporting something like a spread of 7 years of Python > versions. What other platform do you support for 7 years? I think 2.3 will really be gone by the time 2.8 might get released. Even with 2.7, you'd end up with a span of seven years, though. Python had been supporting Windows 95 for more than 7 years (I think rather 9 or 10), likewise Windows 3.1 before that. Python 2.7 will likely still support Windows 2000, which then will be ten years old. Solaris support will probably go back to Solaris 2.6, which will be 13 years old when Python 2.7 gets released. It's only the Linux (and OS X) releases that move so quickly. Regards, Martin From martin at v.loewis.de Tue Jan 12 22:56:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 22:56:55 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> Message-ID: <4B4CF027.4080007@v.loewis.de> > Maybe not, but the Distribute feature is there because IMO the > distutils feature by itself isn't particularily useful. You need to > write your own distutils extensions, in practice, and they are not > trivial. I wouldn't say that. My Django port works with bare distutils (as does Django itself), and it works fine. That distribute had to redo it is only because setuptools *replaces* the build_py command, as does the 2to3 support in distutils. So only if you have a different build_py already, you can't use what is in distutils. Regards, Martin From fuzzyman at voidspace.org.uk Tue Jan 12 23:02:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:02:34 +0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CEF3D.8000107@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> Message-ID: <4B4CF17A.1000503@voidspace.org.uk> On 12/01/2010 21:53, "Martin v. L?wis" wrote: >>> a) telling people that they have to move to 2.6 first actually >>> hurts migration, instead of helping, because it implies to them >>> that they have to drop old versions (e.g. 2.3.) - just because >>> they had *always* dropped old versions before supporting new ones. >>> >> Is it just an implication, or is it reality? >> > That's only the implication. However, this was precisely the dialogue > when talking to Django. If you start with "start supporting 2.6", the > immediate response, without listening further, was, "ok, wait until we > drop 2.3, which will be in Spring 2009" (it has happened by now, IIUC). > > Then explain it to the individual you are talking to, wait for the next > developer of the project step along, and see how he brings up the > very same line of thinking (supporting new versions == dropping support > for old versions). > > I think only part of that comes from the maintenance burden. The other > part is that they *want* to drop support for old versions, so that they > can eventually start using new features (e.g. generator expressions). > So they welcome the requirement to support new versions as an excuse > to drop old ones ("it is obvious that you have to drop 2.3 to support > 3.2"). However, their users then won't let them drop old versions. > > > I agree with Martin that the *perception* is that to use Python 2.6 to help you port to Python 3 you have to be willing to drop support for earlier versions of Python. >> >>> b) IMO, people also don't gain much by first migrating to 2.6. >>> In principle, it gives them the opportunity to get py3k warnings. >>> However, I haven't heard a single positive report where these >>> warnings have actually helped people in porting. Yours is the >>> first report saying that you followed the official guideline, >>> but you didn't state whether doing so actually helped (or whether >>> you just ported to 2.6 because the guideline told you to). >>> >> Python 2.6 has other useful features, which I want to take advantage of >> > I think you are a minority with that, being able to actually use the 2.6 > features already. Many projects can't, as they have to support at least > 2.4 still (so the with statement is right out). > > Well, the IronPython community has almost completely moved over to IronPython 2.6. :-) We tend to ship our Python runtime with our applications though. As it happens I'm now working with CPython on the server (Python 2.5) and IronPython in the browser where we are using 2.6. The new property feature is nice, as is having with without a __future__ import. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From regebro at gmail.com Tue Jan 12 23:03:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Jan 2010 23:03:41 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF027.4080007@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <319e029f1001112249u6104702fhf96457bddb44254a@mail.gmail.com> <4B4CF027.4080007@v.loewis.de> Message-ID: <319e029f1001121403x14ef7ec8x729fb80fa67feaef@mail.gmail.com> On Tue, Jan 12, 2010 at 22:56, "Martin v. L?wis" wrote: >> Maybe not, but the Distribute feature is there because IMO the >> distutils feature by itself isn't particularily useful. You need to >> write your own distutils extensions, in practice, and they are not >> trivial. > > I wouldn't say that. My Django port works with bare distutils (as > does Django itself), and it works fine. > > That distribute had to redo it is only because setuptools *replaces* > the build_py command, as does the 2to3 support in distutils. So only > if you have a different build_py already, you can't use what is in > distutils. Yeah, you are right, I misremembered. The actual additional feature is the support for the test command. Testing under Python 2 and 3 without it is annoying. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Jan 12 23:04:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:04:37 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112000726.GO32351@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> Message-ID: <4B4CF1F5.4050403@v.loewis.de> > [...] >> I've done a fair bit of 3.x porting, and I'm firmly convinced that >> 2.x can do nothing: > [...] >> Inherently, 2.8 can't improve on that. > > I agree that there are limitations like the ones you've listed, but I > disagree with your conclusion. Maybe you assume that it's just as hard > to move to 2.8 (using the py3k backports not available in say 2.5) as it > is to 3.x? Not at all, no. I'd rather expect that code that runs on 2.7 will run on 2.8 unmodified. > But a hypothetical 2.8 would also give people a way to move closer to > py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? > I think it's much more likely that libraries like Twisted can support 2.8 > in the near future than 3.x. Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? > Then, when all of your dependencies (or viable alternatives to those > dependencies) are available for 3.x, you'll have an easier transition if > you can start from a 2.x with fewer differences in features. But you won't *have* fewer differences. Just because your code runs on 2.8 doesn't mean it will stop running on 2.3 (if you have a need for that). This doesn't get you any closer - you can't use any of the 2.8 features as long as you have to support older versions of Python. > Fundamentally the more 2.x can converge on 3.x, the easier it will be > for users to make the leap, because it will be a smaller leap. No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. > The > longer the 2.x series lives, the more these newer 2.x versions like 2.7 > and maybe even 2.8 will be available on common platforms for people to > depend upon as minimum versions, which means that as time goes by they > can depend on a version that's closer to 3.x. No, that's incorrect. Suppose 2.7 is the last 2.x release, to be released in 2010. Then, in 2015, it may be that everybody has migrated to 2.7, which then is a common platform. If you release 2.8 in 2012, then, in 2015, people will be split between 2.7 and 2.8, and so there won't be a common platform before 2017. So stopping 2.x development *earlier* will also give us a common platform earlier. Regareds, Martin From martin at v.loewis.de Tue Jan 12 23:07:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jan 2010 23:07:02 +0100 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: <4B4CF286.5040002@v.loewis.de> > I think it would be interesting to see how people are using the tracker, > or how they want to be using it. For example, there are currently over > 1500 open issues with no stage set, some of which seemingly haven't been > read by anyone at all. Would a properly set stage field save issues from > falling into a black hole? I personally don't think this would be the case. What would help is people who actually *work* on the issues, rather than merely reading them. However, this being open source, there is no way to force it (unless you create an incentive, such as the five-for-one deal). Regards, Martin From python at mrabarnett.plus.com Tue Jan 12 23:10:28 2010 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jan 2010 22:10:28 +0000 Subject: [Python-Dev] regex module Message-ID: <4B4CF354.2050603@mrabarnett.plus.com> Hi all, I'm back on the regex module after doing other things and I'd like your opinion on a number of matters: Firstly, the current re module has a bug whereby it doesn't split on zero-width matches. The BDFL has said that this behaviour should be retained by default in case any existing software depends on it. My question is: should my regex module still do this for Python 3? Speaking personally, I'd like it to behave correctly, and Python 3 is the version where backwards-compatibility is allowed to be broken. Secondly, Python 2 is reaching the end of the line and Python 3 is the future. Should I still release a version that works with Python 2? I'm thinking that it could be confusing if new regex module did zero-width splits correctly in Python 3 but not in Python 2. And also, should I release it only for Python 3 as a 'carrot'? Finally, the module allows some extra backslash escapes, eg \g, in the pattern. Should it treat ill-formed escapes, eg \g, as it would have treated them in the re module? Thanks From michael at voidspace.org.uk Tue Jan 12 23:46:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 22:46:49 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 Message-ID: <4B4CFBD9.1090009@voidspace.org.uk> I presume the email below is about the Windows binary. Does the AMD64 release work on intel 64bit and can we make the wording clearer on the download page? The current description is " Windows AMD64 binary". All the best, Michael -------- Original Message -------- Subject: Download Page - AMD64 Date: Fri, 11 Dec 2009 04:03:16 -0600 From: Thomas Brownback To: webmaster at python.org Should I use the AMD64 version of Python on an Intel 64 chip? I know those 64-bit implementations are very similar, but are they similar enough that your AMD64 will work on Intel? If so, perhaps a note should be placed on that page to help avoid confusion. Explicit being better than implicit and all. Thanks for your time, Thomas -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at bemusement.org Tue Jan 12 23:49:56 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Wed, 13 Jan 2010 09:49:56 +1100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100112224956.GC28576@steerpike.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > But a hypothetical 2.8 would also give people a way to move closer to > > py3k without giving up on using all their 2.x-only dependencies. > > How so? If they use anything that is new in 2.8, they *will* need to > drop support for anything before it, no??? > > > I think it's much more likely that libraries like Twisted can support 2.8 > > in the near future than 3.x. > > Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does > that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. From tjreedy at udel.edu Tue Jan 12 23:51:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 17:51:31 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: On 1/12/2010 5:04 PM, "Martin v. L?wis" wrote: > But you won't *have* fewer differences. Just because your code runs > on 2.8 doesn't mean it will stop running on 2.3 (if you have a need > for that). This doesn't get you any closer - you can't use any of > the 2.8 features as long as you have to support older versions of > Python. > >> Fundamentally the more 2.x can converge on 3.x, the easier it will be >> for users to make the leap, because it will be a smaller leap. > > No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they > likely won't. > >> The >> longer the 2.x series lives, the more these newer 2.x versions like 2.7 >> and maybe even 2.8 will be available on common platforms for people to >> depend upon as minimum versions, which means that as time goes by they >> can depend on a version that's closer to 3.x. > > No, that's incorrect. Suppose 2.7 is the last 2.x release, to be > released in 2010. Then, in 2015, it may be that everybody has migrated > to 2.7, which then is a common platform. > > If you release 2.8 in 2012, then, in 2015, people will be split between > 2.7 and 2.8, and so there won't be a common platform before 2017. Just like people today may need to work with both 2.5 and 2.6, or privately backport 2.6 bugfixes to 2.5. > So stopping 2.x development *earlier* will also give us a common > platform earlier. With years of bug fixes and hence high quality. From tjreedy at udel.edu Wed Jan 13 00:05:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Jan 2010 18:05:12 -0500 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On 1/12/2010 5:10 PM, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. Are you writing a new module with a new name? If so, do you expect it to replace or augment re? (This is the same question as for optparse vs. argparse, which I understand to not yet be decided.) > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, 2.7 stdlib is already out. A new engine should get some community testing before going in the stdlib. Even 3.2 beta is not that far off (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? What does re do with analogous cases? Terry Jan Reedy From david.lyon at preisshare.net Wed Jan 13 00:20:54 2010 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 13 Jan 2010 10:20:54 +1100 (EST) Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4C4589.70906@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <20100111014457.GA5289@arctrix.com> <20100111040507.GA7200@arctrix.com> <20100111052713.GA8211@arctrix.com> <4B4ADED6.5080207@v.loewis.de> <1179.218.214.45.58.1263252162.squirrel@syd-srv02.ezyreg.com> <4B4C4589.70906@gmail.com> Message-ID: <1333.218.214.45.58.1263338454.squirrel@syd-srv02.ezyreg.com> > Nick wrote: >>> This has nothing to do with pushing 3.x, but all with managing >>> available manpower and still providing quality software. >> >> Python 3.x needs more carrots. > > As Guido has said a few times, the gains are far greater for *new* > Python developers than they are for existing ones. Well both you and Guido are most likely 100% correct. :-) > They don't have as rich a 3rd party ecosystem > on Python 3 as they would on Python 2.x at this point in time, but > unlike existing developers they also have the luxury of cherry-picking > just those packages that already have Python 3 support. Most likely. I wouldn't want to say anything to discourage people from going to python 3. In other languages, I have much experience of making the jump to 'a whole new world'. It's unsettling at first, but after that, you suddenly find the new model has a better turbo than the last and you're at the next corner faster than you can think. So it's all good. But I still maintain Python 3.0 needs more carrots. For example, if mercurial or any other cool lib gets added to python 3 (and I can name a few that should go in python 3) then they should be added to python 3 and not to python 2.x They would serve as good carrots. Make fresh the python 3 stdlib and preserve the python 2.x stdlib. I really think we are somewhat short on resources to do what Guido has asked about bringing python up to CPAN level with respect to packages. We're starting a long way back with horses and swords and trying to catch a well fed and greased modern machine.. I'm sure we can get a modern package testbot operational for python. But I wish those who were complaining about the packaging problem so much could throw some money at the problem instead of moaning about it. As is done on other open-source projects. Some organisation would be beneficial. Finding funds so that a small group of people could work on the problem would be a great boost to forward progress. I just think python packaging needs six months of that to repair the years and years of neglect and stagnation. Even if it is only beer and bus fare money. It just needs a temporary shot of adrenalin in the arm.. so to speak.. Regards David From martin at v.loewis.de Wed Jan 13 00:28:14 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 00:28:14 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D058E.4030404@v.loewis.de> > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? "intel 64bit" is as clear as mud. It could mean the "Intel 64" architecture, or it could mean the "IA-64" architecture, both are 64-bit architectures with processors manufactured by Intel. The former is officially called AMD64 (not by Intel, but officially), and our AMD64 binaries run on such processors. The latter is also called "Itanium", and we stopped producing binaries for that architecture a while ago; the AMD64 binaries will *not* run on it. The wording could probably be changed to match the reader's expectation more, most likely, it would also get more incorrect in the process. To make it correct and explicit, an entire paragraph educating users about 64-bit architectures and that there are many of them and that they are mutually incompatible might be required. Most likely, Thomas' processor implements the AMD64 architecture, even though it is manufactured by Intel. A short sentence that he would probably understand (given that he used "Intel 64", not "64bit") is: """The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).""" Regards, Martin From martin at v.loewis.de Wed Jan 13 00:30:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 00:30:27 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <20100112224956.GC28576@steerpike.home.puzzling.org> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> <20100112224956.GC28576@steerpike.home.puzzling.org> Message-ID: <4B4D0613.1010503@v.loewis.de> > I'm not talking about Twisted moving to 3.x (FWIW, I think the only > movement there so far is some patches for some -3 warnings). The > situation I'm describing is a project X that: > > (a) has 2.x-only dependencies, and > (b) would like to be as close as possible to 3.x (because they like > the new features and/or want to be as ready as possible to jump > when (a) is fixed). This assumes that jumping to 3.x is easier if you are closer to it. Please trust me that this is not the case. >> No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they >> likely won't. > > But this is my point. I think they would as an intermediate step to > jumping to 3.x (which also requires dropping 2.5, after all!), if for > some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. No, and no. No: it's not an intermediate step, and no, supporting 3.x does *not* require dropping 2.5. Regards, Martin From fuzzyman at voidspace.org.uk Wed Jan 13 00:40:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:40:35 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D058E.4030404@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> Message-ID: <4B4D0873.5070006@voidspace.org.uk> On 12/01/2010 23:28, "Martin v. L?wis" wrote: > [snip...] > """The binaries for AMD64 will also work on processors that implement > the Intel 64 architecture (formerly EM64T), i.e. the architecture that > Microsoft calls x64, and AMD called x86-64 before calling it AMD64. > They will not work on Intel Itanium Processors (formerly IA-64).""" > > To those not intimately aware of the history and present of 64 bit architecture this sentence would be a great addition - thanks. I'm adding it now as a footnote. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From lists at cheimes.de Wed Jan 13 00:41:04 2010 From: lists at cheimes.de (Christian Heimes) Date: Wed, 13 Jan 2010 00:41:04 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D0890.2030505@cheimes.de> Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". The installer works on all AMD64 compatible Intel CPUs. *scnr* As you most likely know all modern Intel 64bit CPUs are based on AMD's x86-64 design. Only the Itanium family is based on the other Intel 64bit design: IA-64. The name AMD64 was chosen because most Linux and BSD distributions call it so. The name 'AMD64' has caused confusion in the past because users thought that the installer works only on AMD CPUs. How about: * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / X86-64 binary -- does not include source) instead of: * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does not include source) ? Christia From fuzzyman at voidspace.org.uk Wed Jan 13 00:55:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 12 Jan 2010 23:55:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0873.5070006@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D058E.4030404@v.loewis.de> <4B4D0873.5070006@voidspace.org.uk> Message-ID: <4B4D0BD4.5050109@voidspace.org.uk> On 12/01/2010 23:40, Michael Foord wrote: > On 12/01/2010 23:28, "Martin v. L?wis" wrote: >> [snip...] >> """The binaries for AMD64 will also work on processors that implement >> the Intel 64 architecture (formerly EM64T), i.e. the architecture that >> Microsoft calls x64, and AMD called x86-64 before calling it AMD64. >> They will not work on Intel Itanium Processors (formerly IA-64).""" >> > > To those not intimately aware of the history and present of 64 bit > architecture this sentence would be a great addition - thanks. I'm > adding it now as a footnote. > I can add it now to the main download page and existing release pages - but are there changes needed to any scripts to change pages created for *new* releases? All the best, Michael Foord > All the best, > > Michael Foord > >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From python at mrabarnett.plus.com Wed Jan 13 01:22:01 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 00:22:01 +0000 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D1229.70708@mrabarnett.plus.com> Terry Reedy wrote: > On 1/12/2010 5:10 PM, MRAB wrote: >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. > > Are you writing a new module with a new name? If so, do you expect it to > replace or augment re? (This is the same question as for optparse vs. > argparse, which I understand to not yet be decided.) > It's a module called 'regex'. It can be used in place of 're' by using "import regex as re", except for differences such as "\g" being a legal group reference in pattern strings. >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? > > 2.7 is in alpha with no plans for 2.8, so unless you finish real soon, > 2.7 stdlib is already out. A new engine should get some community > testing before going in the stdlib. Even 3.2 beta is not that far off > (8-9 months?) Do *you* want to do the extra work for a 2.x release on PyPI? > >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? > > What does re do with analogous cases? > The 're' module treats r"\g" as "g"; both 're' and 'regex' treat, say, r"\q" as "q". The closest analogue to what I'm asking about is that re treats the ill-formed repeat r"x{1," as a literal, which sort of suggests that r"\g" should be treated as "g", but r"\g" is now a group reference (re would treat that as "g". Does that sound reasonable? From fuzzyman at voidspace.org.uk Wed Jan 13 01:50:39 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 00:50:39 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D18DF.6070502@voidspace.org.uk> On 12/01/2010 23:41, Christian Heimes wrote: > Michael Foord wrote: > >> I presume the email below is about the Windows binary. Does the AMD64 >> release work on intel 64bit and can we make the wording clearer on the >> download page? >> >> The current description is " Windows AMD64 binary". >> > The installer works on all AMD64 compatible Intel CPUs. *scnr* > > As you most likely know all modern Intel 64bit CPUs are based on AMD's > x86-64 design. Only the Itanium family is based on the other Intel 64bit > design: IA-64. The name AMD64 was chosen because most Linux and BSD > distributions call it so. The name 'AMD64' has caused confusion in the > past because users thought that the installer works only on AMD CPUs. > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) > Right - I've made that change for the Python 2.6, 2.7, 3.0 and 3.1 download pages with a footnote. Prior to that we were offering ia64 *and* amd64 releases so I've left those unchanged. Thanks, Michael > ? > > Christia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From exarkun at twistedmatrix.com Wed Jan 13 02:40:03 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 13 Jan 2010 01:40:03 -0000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF1F5.4050403@v.loewis.de> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100112000726.GO32351@steerpike.home.puzzling.org> <4B4CF1F5.4050403@v.loewis.de> Message-ID: <20100113014003.1898.1305834328.divmod.xquotient.219@localhost.localdomain> On 12 Jan, 10:04 pm, martin at v.loewis.de wrote: >>[...] >>>I've done a fair bit of 3.x porting, and I'm firmly convinced that >>>2.x can do nothing: >>[...] >>>Inherently, 2.8 can't improve on that. >> >>I agree that there are limitations like the ones you've listed, but I >>disagree with your conclusion. Maybe you assume that it's just as >>hard >>to move to 2.8 (using the py3k backports not available in say 2.5) as >>it >>is to 3.x? > >Not at all, no. I'd rather expect that code that runs on 2.7 will run >on 2.8 unmodified. >>But a hypothetical 2.8 would also give people a way to move closer to >>py3k without giving up on using all their 2.x-only dependencies. > >How so? If they use anything that is new in 2.8, they *will* need to >drop support for anything before it, no??? >>I think it's much more likely that libraries like Twisted can support >>2.8 >>in the near future than 3.x. > >Most likely, Twisted "supports" 2.8 *today* (hopefully). But how does >that help Twisted in moving to 3.2? I'm not reading this thread carefully enough to make any arguments on either side, but I can contribute a fact. Twisted very likely does not support 2.8 today. I base this on the fact that Twisted does not support 2.7 today, and I expect 2.8 will be more like 2.7 than it will be like 2.3 - 2.6 (which Twisted does support). When I say "support" here, I mean "all of the Twisted unit tests pass on it". Jean-Paul From tseaver at palladion.com Wed Jan 13 03:57:46 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jan 2010 21:57:46 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: <4B4D36AA.7020607@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karen Tracey wrote: > On Sat, Jan 9, 2010 at 12:29 PM, Benjamin Peterson wrote: > >> On behalf of the Python development team, I'm gleeful to announce the >> second >> alpha release of Python 2.7. >> >> > Well yay. Django's test suite (1242 tests) runs with just one failure on > the 2.7 alpha 2 level, and that looks to be likely due to the improved > string/float rounding so not really a problem, just a difference. That's > down from 104 failures and 40 errors with 2.7 alpha 1. > > Note on the website page http://www.python.org/download/releases/2.7/ the > "Change log for this release" link is still pointing to the alpha 1 > changelog. Just to add another "success" data point: Zope2's trunk, as well as the 2.12 release, passes all tests (2535 on the trunk) and brings up the appserver just fine under 2.7a2. There is an obnoxious deprecation warning out of the distutils: DeprecationWarning: 'compiler' specifies the compiler type in build_ext. If you want to get the compiler object itself, use 'compiler_obj' which is likely a simple one-line fix, if I only knew what the hell it was whining about. ;) The warning is extra obnoxious because it doesn't tell me what in *my* code triggers the warning (it (needs a 'stacklevel' argument). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktNNqQACgkQ+gerLs4ltQ7d5gCcCGKVjyOlxnrAln0UnRibS7kd lNIAoIs1RlSGMtJWaY11BqptfDmQvR87 =mIOO -----END PGP SIGNATURE----- From python at mrabarnett.plus.com Wed Jan 13 04:09:34 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Jan 2010 03:09:34 +0000 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: <4B4D396E.4050500@mrabarnett.plus.com> MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > I've just noticed something odd about the re module: the sub() method doesn't take 'pos' or 'endpos' arguments. search() does; match() does; findall() does(); finditer() does; but sub() doesn't. Maybe there has never been a demand for it. (Nor split(), for that matter.) From brett at python.org Wed Jan 13 04:58:11 2010 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2010 19:58:11 -0800 Subject: [Python-Dev] regex module In-Reply-To: <4B4CF354.2050603@mrabarnett.plus.com> References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: On Tue, Jan 12, 2010 at 14:10, MRAB wrote: > Hi all, > > I'm back on the regex module after doing other things and I'd like your > opinion on a number of matters: > > Firstly, the current re module has a bug whereby it doesn't split on > zero-width matches. The BDFL has said that this behaviour should be > retained by default in case any existing software depends on it. My > question is: should my regex module still do this for Python 3? > Speaking personally, I'd like it to behave correctly, and Python 3 is > the version where backwards-compatibility is allowed to be broken. > > If it is a separate module under a different name it can do the proper thing. People will just need to be aware of the difference when they import the module. > Secondly, Python 2 is reaching the end of the line and Python 3 is the > future. Should I still release a version that works with Python 2? I'm > thinking that it could be confusing if new regex module did zero-width > splits correctly in Python 3 but not in Python 2. And also, should I > release it only for Python 3 as a 'carrot'? > > That's totally up to you. There is practically no chance of it getting into the 2.x under the stdlib at this point since 2.7b1 is coming up and this module has not been out in the wild for a year (to my knowledge). If you want to support 2.x that's fine and I am sure users would appreciate it, but it isn't necessary to get into the Python 3 stdlib. > Finally, the module allows some extra backslash escapes, eg \g, in > the pattern. Should it treat ill-formed escapes, eg \g, as it would have > treated them in the re module? > > If you want to minimize the differences then it should probably match. As I said, since it is a different name to import under it can deviate where reasonable, just make sure to clearly document the deviations. -Brett > Thanks > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jan 13 07:11:49 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 07:11:49 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D0890.2030505@cheimes.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4D6425.3060307@v.loewis.de> > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) > > instead of: > > * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does > not include source) -1. AMD doesn't want us to use the term x86-64 anymore, but wants us to use AMD64 instead. I think we should comply - they invented the architecture, so they have the right to give it a name. Neither Microsoft nor Intel have such a right. Regards, Martin From sridharr at activestate.com Wed Jan 13 07:47:38 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 12 Jan 2010 22:47:38 -0800 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4CFBD9.1090009@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> Message-ID: <4B4D6C8A.7070307@activestate.com> On 1/12/2010 2:46 PM, Michael Foord wrote: > I presume the email below is about the Windows binary. Does the AMD64 > release work on intel 64bit and can we make the wording clearer on the > download page? > > The current description is " Windows AMD64 binary". FWIW, we simply use (64-bit, x64). Platform Download ============================================== Windows (x86) Windows Installer (MSI) Windows (64-bit, x64) Windows Installer (MSI) https://www.activestate.com/activepython/downloads/ -srid From solipsis at pitrou.net Wed Jan 13 08:08:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Jan 2010 07:08:55 +0000 (UTC) Subject: [Python-Dev] Fwd: Download Page - AMD64 References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > How about: > > * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / > X86-64 binary -- does not include source) +1. I don't care about trademarks or official names, we should call it whatever is obvious for our users. As for Itanium, it is practically dead, I think there is little chance of people mixing it up with x86-64. cheers Antoine. From michael at voidspace.org.uk Wed Jan 13 10:32:00 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 13 Jan 2010 09:32:00 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4D9310.6060907@voidspace.org.uk> On 13/01/2010 06:11, "Martin v. L?wis" wrote: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) >> >> instead of: >> >> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >> not include source) >> > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. > I think we should use whatever is most informative and least confusing to our users - we owe our allegiance to them and not to a processor vendor. Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Wed Jan 13 11:30:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:30:50 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4CF17A.1000503@voidspace.org.uk> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> Message-ID: <4B4DA0DA.7070007@gmail.com> Michael Foord wrote: > I agree with Martin that the *perception* is that to use Python 2.6 to > help you port to Python 3 you have to be willing to drop support for > earlier versions of Python. Note that increased 3.x compatibility in the most recent 2.x release will always help in two scenarios: 1. New projects that want to use 2.x only libraries but want to be ready for the Py3k transition in their own code (e.g. new 2.7 features like set literals, dict and set comprehensions and the view* dict methods can be translated far more cleanly by 2to3 than the closest comparable 2.6 code). 2. Similarly new projects that use a 3.x trunk can be more readily translated to a 2.7 version with 3to2, whereas some constructs may be difficult to recreate in earlier Python versions. I would expect this to be significantly less common then the first scenario though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:38:31 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:38:31 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> Message-ID: <4B4DA2A7.2080408@gmail.com> Antoine Pitrou wrote: > Christian Heimes cheimes.de> writes: >> How about: >> >> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >> X86-64 binary -- does not include source) > > +1. I don't care about trademarks or official names, we should call it whatever > is obvious for our users. Until this discussion, I didn't even know the architecture had any name other than x86-64. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jan 13 11:39:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Jan 2010 20:39:40 +1000 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4D36AA.7020607@palladion.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> Message-ID: <4B4DA2EC.3050908@gmail.com> Tres Seaver wrote: > There is an obnoxious deprecation warning out of the distutils: > > DeprecationWarning: 'compiler' specifies the compiler type in > build_ext. If you want to get the compiler object itself, use > 'compiler_obj' > > which is likely a simple one-line fix, if I only knew what the hell it > was whining about. ;) The warning is extra obnoxious because it doesn't > tell me what in *my* code triggers the warning (it (needs a 'stacklevel' > argument). Could you kick a tracker issues in Tarek's direction for that one? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Wed Jan 13 13:24:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 13 Jan 2010 12:24:09 +0000 (UTC) Subject: [Python-Dev] topics I plan to discuss at the language summit References: Message-ID: Brett Cannon python.org> writes: > If there is something missing from the stdlib discussion that you think should be brought up at the summit let me know. And if there is something here you want to discuss before the summit let me know and I can start a separate thread on it. I'm sorry I won't be able to come to the language summit, but I would like if possible to expedite a pronouncement on PEP 391 (configuring logging using dictionaries). I believe I addressed all the comments made on the discussion threads mentioned in the PEP and so I'm not sure what more I need to do to get a pronouncement. I guess the stdlib slot gives an opportunity for people to air their views and so I'd be grateful if you added it to the agenda. Thanks & regards, Vinay Sajip From murman at gmail.com Wed Jan 13 14:35:51 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 07:35:51 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D6425.3060307@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: > -1. AMD doesn't want us to use the term x86-64 anymore, but wants us > to use AMD64 instead. I think we should comply - they invented the > architecture, so they have the right to give it a name. Neither > Microsoft nor Intel have such a right. I see and agree with the motivation behind your point, but it's just as reasonable to mimic the name Microsoft uses: the release is for Windows rather than the processor. On MSDN Microsoft uses parentheticals x86, ia64 and x64; this would suggest a name like Python 2.6.4 installer for Windows (x64). Unfortunately this usage doesn't seem to be reflected in consumer-oriented product pages, so I'm uncertain how clear it is for those downloading Python. -- Michael Urman From ncoghlan at gmail.com Wed Jan 13 15:04:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 00:04:28 +1000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> Message-ID: <4B4DD2EC.3030709@gmail.com> Michael Urman wrote: > On Wed, Jan 13, 2010 at 00:11, "Martin v. L?wis" wrote: >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. > > I see and agree with the motivation behind your point, but it's just > as reasonable to mimic the name Microsoft uses: the release is for > Windows rather than the processor. On MSDN Microsoft uses > parentheticals x86, ia64 and x64; this would suggest a name like > Python 2.6.4 installer for Windows (x64). Unfortunately this usage > doesn't seem to be reflected in consumer-oriented product pages, so > I'm uncertain how clear it is for those downloading Python. As Windows doesn't run on non-x86 architectures, their naming is generally just Windows (32 bit) and Windows (64 bit). Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. In this case, making it clear that the 64-bit Windows binaries work for both Intel and AMD chips is more important than using only the official architecture name. Cheers, Nick. P.S. This discussion made me realise that out of habit I had dropped the 32-bit version of Python on my new 64-bit Windows box... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tseaver at palladion.com Wed Jan 13 17:49:55 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 13 Jan 2010 11:49:55 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA2EC.3050908@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4D36AA.7020607@palladion.com> <4B4DA2EC.3050908@gmail.com> Message-ID: <4B4DF9B3.4030403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > Tres Seaver wrote: >> There is an obnoxious deprecation warning out of the distutils: >> >> DeprecationWarning: 'compiler' specifies the compiler type in >> build_ext. If you want to get the compiler object itself, use >> 'compiler_obj' >> >> which is likely a simple one-line fix, if I only knew what the hell it >> was whining about. ;) The warning is extra obnoxious because it doesn't >> tell me what in *my* code triggers the warning (it (needs a 'stacklevel' >> argument). > > Could you kick a tracker issues in Tarek's direction for that one? Done: http://bugs.python.org/issue7694 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktN+bMACgkQ+gerLs4ltQ6s4gCdENhtVQWzoH449BCDz8SNx/4s DEoAn19LMcMs3b6ctdNF8K2hYd6d+BSZ =TWit -----END PGP SIGNATURE----- From rdmurray at bitdance.com Wed Jan 13 18:24:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Jan 2010 12:24:42 -0500 Subject: [Python-Dev] PYTHON3PATH Message-ID: <20100113172442.4A7771FA679@kimball.webabinitio.net> Please review issue 2375 [1], which is an enhancement request to add a PYTHON3PATH environment variable. Because we have elected to have both a python and a python3 command, I think this is an issue worth thinking about carefully to make sure we are serving the Python user community and easing the transition to python3. It could be that "use virtualenv" is the best answer, but I feel we should think about it carefully to make sure that is really true. [1] http://bugs.python.org/issue2375 -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From guido at python.org Wed Jan 13 18:31:39 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:31:39 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: Somehow the bug site doesn't load for me right now, but I'm -1 on this. There are maybe a dozen PYTHON... variables -- we really shouldn't try to add PYTHON3 variants for all of them. Specifically for PYTHONPATH, I feel that its use is always a short-time or localized hack, not something you set in your .profile nd forget about it (forgetting about it inparticular often leads to unnecessary debugging pain later). The better approaches are based on site-packages, wrapper scripts, virtualenv, etc. --Guido On Wed, Jan 13, 2010 at 9:24 AM, R. David Murray wrote: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. ?Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. ?It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 > > -- > R. David Murray ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?www.bitdance.com > Business Process Automation - Network/Server Management - Routers/Firewalls > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From dirkjan at ochtman.nl Wed Jan 13 18:38:26 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 13 Jan 2010 18:38:26 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> Message-ID: On Sat, Jan 9, 2010 at 18:29, Benjamin Peterson wrote: > Please consider trying Python 2.7 with your code and reporting any bugs you may I ran the Mercurial test suite on current trunk, and it worked alright. There was a small issue with the fact that mimetypes got fooled by the lack of ImportError from _winreg (which I solved by blacklisting _winreg in demandimport), and one test fails likely because of a change in MIME handling. Will look into that. So the state of trunk looks rather solid from where I sit. Cheers, Dirkjan From ralf at brainbot.com Wed Jan 13 18:40:04 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:40:04 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113172442.4A7771FA679@kimball.webabinitio.net> (R. David Murray's message of "Wed, 13 Jan 2010 12:24:42 -0500") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87r5ptdhu3.fsf@muni.brainbot.com> "R. David Murray" writes: > Please review issue 2375 [1], which is an enhancement request to add a > PYTHON3PATH environment variable. Because we have elected to have both > a python and a python3 command, I think this is an issue worth thinking > about carefully to make sure we are serving the Python user community > and easing the transition to python3. It could be that "use virtualenv" > is the best answer, but I feel we should think about it carefully to > make sure that is really true. > > [1] http://bugs.python.org/issue2375 The first thing I got while trying to run a python3 prompt few days ago, was an error. python3 tried to read my $PYTHONSTARTUP file, which used print statements. people will have to run both python 2 and python 3 code at the same time. Using different environment variables will make this easier. - Ralf From ralf at brainbot.com Wed Jan 13 18:55:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 18:55:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Guido van Rossum's message of "Wed, 13 Jan 2010 09:31:39 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> Message-ID: <87k4vldh4c.fsf@muni.brainbot.com> Guido van Rossum writes: > Somehow the bug site doesn't load for me right now, but I'm -1 on > this. There are maybe a dozen PYTHON... variables -- we really > shouldn't try to add PYTHON3 variants for all of them. > > Specifically for PYTHONPATH, I feel that its use is always a > short-time or localized hack, not something you set in your .profile > nd forget about it (forgetting about it inparticular often leads to > unnecessary debugging pain later). The better approaches are based on > site-packages, wrapper scripts, virtualenv, etc. then at least remove them completely from python3, so one can use them for his python 2 installation. Otherwise it will stump on some innocent python 3 program (and cause unnecessary debugging pain). From steven.bethard at gmail.com Wed Jan 13 18:57:29 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 13 Jan 2010 09:57:29 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: On Wed, Jan 13, 2010 at 9:40 AM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. ?Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. ?It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. How complicated is your PYTHONSTARTUP file? My suspicion is that you could easily write it to work for both Python 2.X and 3.X. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ssteinerx at gmail.com Wed Jan 13 18:57:42 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 13 Jan 2010 12:57:42 -0500 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> On Jan 13, 2010, at 12:40 PM, Ralf Schmitt wrote: > "R. David Murray" writes: > >> Please review issue 2375 [1], which is an enhancement request to add a >> PYTHON3PATH environment variable. Because we have elected to have both >> a python and a python3 command, I think this is an issue worth thinking >> about carefully to make sure we are serving the Python user community >> and easing the transition to python3. It could be that "use virtualenv" >> is the best answer, but I feel we should think about it carefully to >> make sure that is really true. >> >> [1] http://bugs.python.org/issue2375 > > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. Python 2 will continue to run as it has, and better solutions will emerge for Python 3 development. Beats the heck out of making a whole duplicate set for PYTHON3BLAHBLAH, yuck! S From guido at python.org Wed Jan 13 18:58:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 09:58:04 -0800 Subject: [Python-Dev] regex module In-Reply-To: References: <4B4CF354.2050603@mrabarnett.plus.com> Message-ID: Memories of days past... Python had several regular expression implementations before, one of which was called "regex". But I would rather not have a new module -- I would much rather have a flag specifying the new (backwards incompatible) syntax/semantics. The flag would have a long name (e.g. re.NEW_SYNTAX), a short name (e.g. re.N) and an inline syntax, "(?n)...". --Guido On Tue, Jan 12, 2010 at 7:58 PM, Brett Cannon wrote: > > > On Tue, Jan 12, 2010 at 14:10, MRAB wrote: >> >> Hi all, >> >> I'm back on the regex module after doing other things and I'd like your >> opinion on a number of matters: >> >> Firstly, the current re module has a bug whereby it doesn't split on >> zero-width matches. The BDFL has said that this behaviour should be >> retained by default in case any existing software depends on it. My >> question is: should my regex module still do this for Python 3? >> Speaking personally, I'd like it to behave correctly, and Python 3 is >> the version where backwards-compatibility is allowed to be broken. >> > > If it is a separate module under a different name it can do the proper > thing. People will just need to be aware of the difference when they import > the module. > >> >> Secondly, Python 2 is reaching the end of the line and Python 3 is the >> future. Should I still release a version that works with Python 2? I'm >> thinking that it could be confusing if new regex module did zero-width >> splits correctly in Python 3 but not in Python 2. And also, should I >> release it only for Python 3 as a 'carrot'? >> > > That's totally up to you. There is practically no chance of it getting into > the 2.x under the stdlib at this point since 2.7b1 is coming up and this > module has not been out in the wild for a year (to my knowledge). ?If you > want to support 2.x that's fine and I am sure users would appreciate it, but > it isn't necessary to get into the Python 3 stdlib. > >> >> Finally, the module allows some extra backslash escapes, eg \g, in >> the pattern. Should it treat ill-formed escapes, eg \g, as it would have >> treated them in the re module? >> > > If you want to minimize the differences then it should probably match. As I > said, since it is a different name to import under it can deviate where > reasonable, just make sure to clearly document the deviations. > -Brett > >> >> Thanks >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From ralf at brainbot.com Wed Jan 13 19:03:08 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 19:03:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: (Steven Bethard's message of "Wed, 13 Jan 2010 09:57:29 -0800") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <87fx69dgrn.fsf@muni.brainbot.com> Steven Bethard writes: > > How complicated is your PYTHONSTARTUP file? My suspicion is that you > could easily write it to work for both Python 2.X and 3.X. sure. that's exactly what I did. My point is that sharing those environment variables will cause pain for some people. From guido at python.org Wed Jan 13 19:07:58 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:07:58 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about just removing the antiquated use of environment variables altogether from Python 3 and avoid the issue completely. -1. They have their use, but more in controlled situations. If you have "global" env vars that you only want to use with Python 2.x, write a wrapper for Python 3 that invokes it with -E. -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Wed Jan 13 19:14:21 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 13 Jan 2010 13:14:21 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E0D7D.3040806@scottdial.com> On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Wed Jan 13 19:22:33 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:22:33 -0800 Subject: [Python-Dev] topics I plan to discuss at the language summit In-Reply-To: References: Message-ID: On Wed, Jan 13, 2010 at 4:24 AM, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> If there is something missing from the stdlib discussion that you think should > be brought up at the summit let me know. And if there is something here you want > to discuss before the summit let me know and I can start a separate thread on it. > > I'm sorry I won't be able to come to the language summit, but I would like if > possible to expedite a pronouncement on PEP 391 (configuring logging using > dictionaries). I believe I addressed all the comments made on the discussion > threads mentioned in the PEP and so I'm not sure what more I need to do to get a > pronouncement. I guess the stdlib slot gives an opportunity for people to air > their views and so I'd be grateful if you added it to the agenda. Is the PEP in the stage of consensus yet? If so it may not even be necessary to discuss it at the summit (though I certainly won't stop people if they want to). -- --Guido van Rossum (python.org/~guido) From phd at phd.pp.ru Wed Jan 13 19:18:50 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 21:18:50 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <20100113181850.GA3837@phd.pp.ru> On Wed, Jan 13, 2010 at 12:57:42PM -0500, ssteinerX at gmail.com wrote: > Or, how about just removing the antiquated use of environment variables "antiquated"? You are kidding, aren't you?! Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Jan 13 19:51:14 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 10:51:14 -0800 Subject: [Python-Dev] PyCon Keynote Message-ID: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Jan 13 20:13:58 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:13:58 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4D9310.6060907@voidspace.org.uk> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> Message-ID: <4B4E1B76.4010309@v.loewis.de> >>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>> X86-64 binary -- does not include source) >>> >>> instead of: >>> >>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>> not include source) >>> >> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >> to use AMD64 instead. I think we should comply - they invented the >> architecture, so they have the right to give it a name. Neither >> Microsoft nor Intel have such a right. >> > > I think we should use whatever is most informative and least confusing > to our users - we owe our allegiance to them and not to a processor vendor. And why do you think this is x86-64? Regards, Martin From martin at v.loewis.de Wed Jan 13 20:33:24 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:33:24 +0100 Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 2 In-Reply-To: <4B4DA0DA.7070007@gmail.com> References: <1afaf6161001090929x228deda5id07cc762522ca53e@mail.gmail.com> <4B4A373F.9050601@gmail.com> <4B4A5DCE.3070909@v.loewis.de> <4B4B9B55.1010709@v.loewis.de> <20100111191128.39020d89@freewill> <4B4CEF3D.8000107@v.loewis.de> <4B4CF17A.1000503@voidspace.org.uk> <4B4DA0DA.7070007@gmail.com> Message-ID: <4B4E2004.8050905@v.loewis.de> > Note that increased 3.x compatibility in the most recent 2.x release > will always help in two scenarios: > > 1. New projects that want to use 2.x only libraries but want to be ready > for the Py3k transition in their own code (e.g. new 2.7 features like > set literals, dict and set comprehensions and the view* dict methods can > be translated far more cleanly by 2to3 than the closest comparable 2.6 > code). Of these, I can only see this being the case of the view* methods. Why would set literals allow for code that more cleanly translates to 3.x? Suppose you write f = set((1,2,3)) in 2.6. That code will work *exactly* the same way in 3.1, no translation needed at all. Likewise for dict and set comprehensions: the code is as cleanly translated in the 2.7 form as it is in any prior form (ie. no modifications). As for view* methods: there is one important exception where the 2.6 equivalent is as cleanly translated as a view method: for key in D: action This is a more modern equivalent for iterating over D.keys(), so if your code still does the latter, just change it to the former (requires Python 2.2). I'd claim that this is the dominant case of traversing a dictionary (probably followed by iterating over .items()). So while it is true that only view* can be transformed correctly, I'd claim that this is an infrequent issue. > 2. Similarly new projects that use a 3.x trunk can be more readily > translated to a 2.7 version with 3to2, whereas some constructs may be > difficult to recreate in earlier Python versions. That may be true, but is besides the point here: the issue was whether a 2.8 release might help in migrating to 3.x. People who follow this approach already have 3.x code. Whether they would rather port to 2.8 only, and get that work with little effort, or whether they would port to 2.5 and later, and put in more work, I don't know - I guess we would have to ask these people. I would expect that they prefer if 2.x died rather sooner than later, because they can then stop supporting it. Regards, Martin From tjreedy at udel.edu Wed Jan 13 20:36:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Jan 2010 14:36:03 -0500 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: On 1/13/2010 2:13 PM, "Martin v. L?wis" wrote: >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. > > And why do you think this is x86-64? I do not believe I have personally seen, or at least noticed, that specific term before. Something like "Release for 64-bit Windows (Win NT, 2000, XP, Vista, 7 ) on AMD64-type processors from AMD and Intel" should be clear enough. From martin at v.loewis.de Wed Jan 13 20:40:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 13 Jan 2010 20:40:30 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4DD2EC.3030709@gmail.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> Message-ID: <4B4E21AE.40602@v.loewis.de> > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). Windows actually does - it runs on IA-64 (which is a non-x86 architecture). However, end users are unlikely to use such hardware, so distinguishing between 32-bit and 64-bit is typically good enough. > In this case, making it clear that the 64-bit Windows binaries work for > both Intel and AMD chips is more important than using only the official > architecture name. Well, we did have Itanium binaries at one point, and the "64-bit Windows binaries" you are referring to most definitely don't run on Itanium, even though that's an Intel chip. Regards, Martin From martin at v.loewis.de Wed Jan 13 20:45:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 20:45:40 +0100 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E0D7D.3040806@scottdial.com> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> Message-ID: <4B4E22E4.4040504@v.loewis.de> > So to echo what Michael said, the Microsoft nomenclature is "x64" > regardless of yours and Martin's objections to that name. Nobody who > uses Windows would be confused by "x64" since that is *the* Microsoft > naming scheme. That's actually not entirely true. There are several places in the APIs where Microsoft either allows or requires to call the architecture AMD64 (e.g. architecture indication in MSI files). Regards, Martin From regebro at gmail.com Wed Jan 13 20:50:59 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 20:50:59 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87r5ptdhu3.fsf@muni.brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> Message-ID: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: > The first thing I got while trying to run a python3 prompt few days ago, > was an error. python3 tried to read my $PYTHONSTARTUP file, which used > print statements. people will have to run both python 2 and python 3 > code at the same time. Using different environment variables will make > this easier. What do you need to do in the PYTHONSTARTUP file? Ten years of Python programming, and I didn't even know it existed. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From phd at phd.pp.ru Wed Jan 13 21:08:40 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 13 Jan 2010 23:08:40 +0300 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <20100113200840.GC14858@phd.pp.ru> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From murman at gmail.com Wed Jan 13 21:12:11 2010 From: murman at gmail.com (Michael Urman) Date: Wed, 13 Jan 2010 14:12:11 -0600 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E22E4.4040504@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4DD2EC.3030709@gmail.com> <4B4E0D7D.3040806@scottdial.com> <4B4E22E4.4040504@v.loewis.de> Message-ID: On Wed, Jan 13, 2010 at 13:45, "Martin v. L?wis" wrote: >> So to echo what Michael said, the Microsoft nomenclature is "x64" >> regardless of yours and Martin's objections to that name. Nobody who >> uses Windows would be confused by "x64" since that is *the* Microsoft >> naming scheme. > > That's actually not entirely true. There are several places in the > APIs where Microsoft either allows or requires to call the architecture > AMD64 (e.g. architecture indication in MSI files). I should have clarified I was talking about the names shown on MSDN subscriptions for downloading installation media of Windows 7 and Windows Vista. It's pretty clear in that context Microsoft uses x64 to describe this platform of Windows. But again, it's far from clear that this is a term they use for non-developers. -- Michael Urman From regebro at gmail.com Wed Jan 13 21:27:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Jan 2010 21:27:08 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <20100113200840.GC14858@phd.pp.ru> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> Message-ID: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: > On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >> What do you need to do in the PYTHONSTARTUP file? >> Ten years of Python programming, and I didn't even know it existed. :-) > > ? See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. Cheese and fries! :-) Anyway, I don't see how this could pose any problems, because any user advanced enough to do these things will be advanced enough to understand what the problem is and fix it so it works under Python 3 too. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ralf at brainbot.com Wed Jan 13 21:52:34 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 13 Jan 2010 20:52:34 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> (Lennart Regebro's message of "Wed, 13 Jan 2010 20:50:59 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <874ompg225.fsf@brainbot.com> Lennart Regebro writes: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? > Ten years of Python programming, and I didn't even know it existed. :-) hehe. tab completion: # -*- mode: python -*- # Last changed: 2009-12-23 22:25:15 by ralf import sys import os def initreadline(): try: import readline except ImportError: sys.stdout.write("Module readline not available.\n") return import rlcompleter readline.parse_and_bind("tab: complete") # Use tab for completions readline.parse_and_bind('tab: complete') # This forces readline to automatically print the above list when tab # completion is set to 'complete'. readline.parse_and_bind('set show-all-if-ambiguous on') # Bindings for incremental searches in the history. These searches # use the string typed so far on the command line and search # anything in the previous input history containing them. readline.parse_and_bind('"\C-r": reverse-search-history') readline.parse_and_bind('"\C-s": forward-search-history') history = os.path.expanduser("~/.pyhistory") if os.path.exists(history): readline.read_history_file(history) import atexit atexit.register(lambda: readline.write_history_file(history)) initreadline() del initreadline From martin at v.loewis.de Wed Jan 13 22:05:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Jan 2010 22:05:03 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <4B4E357F.4050107@v.loewis.de> Lennart Regebro wrote: > On Wed, Jan 13, 2010 at 18:40, Ralf Schmitt wrote: >> The first thing I got while trying to run a python3 prompt few days ago, >> was an error. python3 tried to read my $PYTHONSTARTUP file, which used >> print statements. people will have to run both python 2 and python 3 >> code at the same time. Using different environment variables will make >> this easier. > > What do you need to do in the PYTHONSTARTUP file? I personally set up readline: set tab completion, and load the history of commands from the previous session. Of course, I don't know what Ralf uses it for. Regards, Martin From ncoghlan at gmail.com Wed Jan 13 22:43:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 07:43:41 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> Message-ID: <4B4E3E8D.2010407@gmail.com> Guido van Rossum wrote: > On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about > just removing the antiquated use of environment variables altogether > from Python 3 and avoid the issue completely. > > -1. They have their use, but more in controlled situations. If you > have "global" env vars that you only want to use with Python 2.x, > write a wrapper for Python 3 that invokes it with -E. Perhaps a case can be made for Python 3 to assume -E by default, with a -e option to enable reading of the environment variables? That way naive users could run Python3 without tripping over existing Python2 environment variables, while other tools could readily set up a different environment before launching Python3. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Wed Jan 13 23:45:57 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Jan 2010 14:45:57 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: On Wed, Jan 13, 2010 at 1:43 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users using environment variables? That's a recipe for disaster in any version! :-) -- --Guido van Rossum (python.org/~guido) From jared.grubb at gmail.com Wed Jan 13 23:56:37 2010 From: jared.grubb at gmail.com (Jared Grubb) Date: Wed, 13 Jan 2010 14:56:37 -0800 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <13F6C43A-D62A-4A9F-AF7A-9BDAC94F7D72@gmail.com> On 13 Jan 2010, at 13:43, Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. I raised a question about these PYTHON3* variables once before in a discussion about shebang lines: http://www.mailinglistarchive.com/python-dev at python.org/msg29500.html I'm not advocating them, but just wanted to make sure to bring up the shebang use case. Jared From skip at pobox.com Wed Jan 13 23:24:13 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 16:24:13 -0600 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> Message-ID: <19278.18445.428716.321365@montanaro.dyndns.org> Lennart> What do you need to do in the PYTHONSTARTUP file? Just reading off stuff from my own personal startup file... I use it for stuff I want available during interactive sessions: 1. Enable true division. 2. Conditionally define "help" from back in the days when there was no help builtin function. 3. Auto-save session (readline) history so I can easily recall commands across sessions. 4. Add other fake builtins ("like") or override behavior of some (like "dir") making them handier for interactive use. 5. autoload modules/symbols (pokes around in common modules from sys.excepthook function). Oh, and I've had no particular trouble keeping it working in Python 1, 2 or 3. Skip From fuzzyman at voidspace.org.uk Thu Jan 14 01:20:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 00:20:21 +0000 Subject: [Python-Dev] Fwd: Download Page - AMD64 In-Reply-To: <4B4E1B76.4010309@v.loewis.de> References: <4B4CFBD9.1090009@voidspace.org.uk> <4B4D0890.2030505@cheimes.de> <4B4D6425.3060307@v.loewis.de> <4B4D9310.6060907@voidspace.org.uk> <4B4E1B76.4010309@v.loewis.de> Message-ID: <4B4E6345.5070202@voidspace.org.uk> On 13/01/2010 19:13, "Martin v. L?wis" wrote: >>>> * Python 2.6.4 Windows X86-64 installer (Windows AMD64 / Intel 64 / >>>> X86-64 binary -- does not include source) >>>> >>>> instead of: >>>> >>>> * Python 2.6.4 Windows AMD64 installer (Windows AMD64 binary -- does >>>> not include source) >>>> >>>> >>> -1. AMD doesn't want us to use the term x86-64 anymore, but wants us >>> to use AMD64 instead. I think we should comply - they invented the >>> architecture, so they have the right to give it a name. Neither >>> Microsoft nor Intel have such a right. >>> >>> >> I think we should use whatever is most informative and least confusing >> to our users - we owe our allegiance to them and not to a processor vendor. >> > And why do you think this is x86-64? > Well anecdotal everyone I have *every* talked to about 64bit processors has referred to having a 64bit processor (x86 is a given) and not an AMD64 architecture processor. Linus Torvalds addressed this specific issue for Linux and came down on the side of "x86-64": http://kerneltrap.org/node/2466 Look up AMD64 on Wikipedia and it redirects you to the X86-64 page. Information website setup by AMD and partners about the AMD64 architecture: http://www.x86-64.org/about.html In the AMD website they refer to "x86-64 Assembly": http://www.x86-64.org/documentation/assembly.html Microsoft seem to draw a distinction between x64 (which would also be acceptable) and Itanium based systems. Very rarely do they refer to AMD64: * http://www.microsoft.com/servers/64bit/compare.mspx * http://www.microsoft.com/servers/64bit/x64/overview.mspx * http://www.microsoft.com/servers/64bit/overview.mspx Using a vendor specific name automatically begs the question as to whether the installer works on processors from other vendors, as we saw in the specific enquiry from the user that triggered this debate. Referring to the AMD 64 build as x86-64, with a footnote explaining which architectures this specifically means is unlikely to confuse people. It is *definitely* better than just saying AMD64. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From skip at pobox.com Thu Jan 14 02:50:55 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 13 Jan 2010 19:50:55 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? Message-ID: <19278.30847.649228.115514@montanaro.dyndns.org> Just out of curiosity, is the static build stuff (use the old Modules/Setup file to build modules) exercised at all any more? Skip From benjamin at python.org Thu Jan 14 03:22:03 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 13 Jan 2010 20:22:03 -0600 Subject: [Python-Dev] static (Modules/Setup) builds? In-Reply-To: <19278.30847.649228.115514@montanaro.dyndns.org> References: <19278.30847.649228.115514@montanaro.dyndns.org> Message-ID: <1afaf6161001131822r151bd202x35653ba44ee0235d@mail.gmail.com> 2010/1/13 : > > Just out of curiosity, is the static build stuff (use the old Modules/Setup > file to build modules) exercised at all any more? Exercised as in used in the build process? Yes. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Jan 14 10:23:47 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 09:23:47 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! Message-ID: <954450.26617.qm@web25804.mail.ukl.yahoo.com> In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: http://www.python.org/dev/peps/pep-0391/ In November 2009 I posted to this list that the PEP was ready for review. I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. Thanks & regards, Vinay Sajip From ziade.tarek at gmail.com Thu Jan 14 10:30:15 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Jan 2010 10:30:15 +0100 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <94bdd2611001140130u6154e893jfd37db32a25be66a@mail.gmail.com> On Thu, Jan 14, 2010 at 10:23 AM, Vinay Sajip wrote: [..] > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would > be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check > in the changes unless there are objections to doing so. All those who think that logging is currently > hard to configure will benefit from these changes, so here's the opportunity to pipe up and > improve things. FWIW, I am +1. Thanks for your work Tarek From hodgestar+pythondev at gmail.com Thu Jan 14 10:39:41 2010 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 14 Jan 2010 11:39:41 +0200 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: On Thu, Jan 14, 2010 at 11:23 AM, Vinay Sajip wrote: > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I think the dict configuration scheme will be a great addition to the logging module. :) Schiavo Simon From p.f.moore at gmail.com Thu Jan 14 12:22:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jan 2010 11:22:09 +0000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> 2010/1/14 Vinay Sajip : > So, can you please indicate your vote for or against incorporating PEP 391 into Python? I've no immediate need for the feature, but it would be good to have something like this, so I'm +1. Paul. From ncoghlan at gmail.com Thu Jan 14 12:46:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:46:19 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <79990c6b1001140322j47fe2030ree2f1cebbc169108@mail.gmail.com> Message-ID: <4B4F040B.8010607@gmail.com> Paul Moore wrote: > 2010/1/14 Vinay Sajip : >> So, can you please indicate your vote for or against incorporating PEP 391 into Python? > > I've no immediate need for the feature, but it would be good to have > something like this, so I'm +1. I'm in the same boat as Paul, but PEP 291 provides a nice forward compatible configuration scheme that will work with any application configuration approach that can produce an appropriate Python dictionary. So +1 from me too - I think the PEP has now taken this as far as theorising can go, and the only way to learn anything further is to put it into practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jan 14 12:57:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Jan 2010 21:57:55 +1000 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4B4F06C3.7030806@gmail.com> Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of > configuring logging using dictionaries: > > http://www.python.org/dev/peps/pep-0391/ Although one minor comment: you can probably remove the note about the "ext://" and "cfg://" prefixes being provisional at this stage. Those names look fine to me, so I don't see any point inviting a late-breaking bikeshed discussion about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu Jan 14 14:53:29 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 08:53:29 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <954450.26617.qm@web25804.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> Message-ID: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> On Thu, Jan 14, 2010 at 4:23 AM, Vinay Sajip wrote: > In October 2009 I created PEP 391 to propose a new method of configuring logging using dictionaries: > > ?http://www.python.org/dev/peps/pep-0391/ > > In November 2009 I posted to this list that the PEP was ready for review. > > I have had numerous helpful comments from some of you, and I have incorporated most of them into updates to the PEP. Though I have the feeling from community discussions that the PEP has been generally favourably received - well I would think that, wouldn't I? ;-) - I've not asked for a vote and so I don't know the state of community consensus regarding this PEP. > > So, can you please indicate your vote for or against incorporating PEP 391 into Python? It would be nice if I could incorporate the changes before 2.7a3 is released! Ideally, I would like to check in the changes unless there are objections to doing so. All those who think that logging is currently hard to configure will benefit from these changes, so here's the opportunity to pipe up and improve things. > > Thanks & regards, > > Vinay Sajip I'm generally +1 - but given I know that Django 1.2 is slated to implement something somewhat similar, I'm interested to hear how this proposal meshes with their plan(s). jesse From vinay_sajip at yahoo.co.uk Thu Jan 14 15:08:24 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jan 2010 14:08:24 +0000 (GMT) Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> Message-ID: <92210.91656.qm@web25806.mail.ukl.yahoo.com> > From: Jesse Noller > I'm generally +1 - but given I know that Django 1.2 is slated to > implement something somewhat similar, I'm interested to hear how this > proposal meshes with their plan(s).. Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: http://groups.google.com/group/django-developers/msg/4ef81a2160257221 They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. Regards, Vinay Sajip From ssteinerx at gmail.com Thu Jan 14 15:54:27 2010 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 14 Jan 2010 09:54:27 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <62E362ED-5DD7-4D42-B5E0-B263A137FD5C@gmail.com> On Jan 14, 2010, at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. That puts a huge +1 on there for me. If we can get this approved and have Django's logging improved *and* avoid a whole pile of duplicate effort in Django that's a huge win. S From jnoller at gmail.com Thu Jan 14 15:56:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 14 Jan 2010 09:56:18 -0500 Subject: [Python-Dev] PEP 391 - Please Vote! In-Reply-To: <92210.91656.qm@web25806.mail.ukl.yahoo.com> References: <954450.26617.qm@web25804.mail.ukl.yahoo.com> <4222a8491001140553o5b91af27sd0e8402bcfca49e7@mail.gmail.com> <92210.91656.qm@web25806.mail.ukl.yahoo.com> Message-ID: <4222a8491001140656w68c7290agccfe7282cf96f2fb@mail.gmail.com> On Thu, Jan 14, 2010 at 9:08 AM, Vinay Sajip wrote: >> From: Jesse Noller > >> I'm generally +1 - but given I know that Django 1.2 is slated to >> implement something somewhat similar, I'm interested to hear how this >> proposal meshes with their plan(s).. > > Django 1.2 will most likely not implement logging - they're now in feature freeze for big features. See Russ Magee's post: > > http://groups.google.com/group/django-developers/msg/4ef81a2160257221 > > They're waiting for a Python decision and (from Russ Magee's post) seem to be in favour of the changes proposed in PEP 391. If we get these changes into Python soon, then Django 1.3 may be able to make use of them in its logging (which encompasses not just configuring Django logging in a settings.py, but also using logging internally throughout Django where it makes sense to do so). > > Assuming PEP 391 gets the nod, then after implementing the changes into Python, I plan to work with the Django community to get improved logging support in Django for 1.3. > > Regards, > > Vinay Sajip > Cool, +1 then :) From mal at egenix.com Thu Jan 14 20:19:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 20:19:12 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4E3E8D.2010407@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> Message-ID: <4B4F6E30.50400@egenix.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >> just removing the antiquated use of environment variables altogether >> from Python 3 and avoid the issue completely. >> >> -1. They have their use, but more in controlled situations. If you >> have "global" env vars that you only want to use with Python 2.x, >> write a wrapper for Python 3 that invokes it with -E. > > Perhaps a case can be made for Python 3 to assume -E by default, with a > -e option to enable reading of the environment variables? > > That way naive users could run Python3 without tripping over existing > Python2 environment variables, while other tools could readily set up a > different environment before launching Python3. Naive users won't have PYTHONPATH or any other Python environment variables setup. Really, if you know that you are going to run Python 3 instead of Python 2 or vice-versa it's easy enough to run . py3env.sh or . py2env.sh in order to setup your shell environment, much like you typically do when using virtual Python installations or work on different projects that require different setups. If you just want to separate Python 2 and 3 files, use the user site-packages dir which includes the Python version. More experienced users could also write their own environment switching sitecustomize.py or usercustomize.py. And then there's the hackery option for experts that love dirty tricks: add a .pth file which includes an "import mysetup" line to fix-up you path and other settings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jan 14 22:02:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Jan 2010 07:02:28 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <4B4F8664.4080107@gmail.com> M.-A. Lemburg wrote: > Nick Coghlan wrote: >> Guido van Rossum wrote: >>> On Wed, Jan 13, 2010 at 9:57 AM, ssteinerX at gmail.com > Or, how about >>> just removing the antiquated use of environment variables altogether >>> from Python 3 and avoid the issue completely. >>> >>> -1. They have their use, but more in controlled situations. If you >>> have "global" env vars that you only want to use with Python 2.x, >>> write a wrapper for Python 3 that invokes it with -E. >> Perhaps a case can be made for Python 3 to assume -E by default, with a >> -e option to enable reading of the environment variables? >> >> That way naive users could run Python3 without tripping over existing >> Python2 environment variables, while other tools could readily set up a >> different environment before launching Python3. > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. I was actually thinking of the situation where the OS had a preinstalled Python 2 with a default PYTHONPATH setting and the user was playing with Python 3. However, I agree that that is a fairly unlikely scenario (since preinstalled Pythons tend not to rely on the environment variables and a user sophisticated enough to be playing with a new Python interpreter should be able to cope with a few environment variable conflicts anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Thu Jan 14 22:09:30 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 14 Jan 2010 21:09:30 +0000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F8664.4080107@gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <4B4F8664.4080107@gmail.com> Message-ID: <4B4F880A.5010809@voidspace.org.uk> On 14/01/2010 21:02, Nick Coghlan wrote: > However, I agree that that is a fairly unlikely scenario (since > preinstalled Pythons tend not to rely on the e Well, on the other hand I think that during the next few years it will be increasingly common for developers (and possibly users) to have Python 2 and Python 3 installed side-by-side. Many libraries and applications may never make the jump to Python 3 and Python users may be using 'legacy' Python 2 code for many years to come. It will also become increasingly common for developers to be using Python 3 *primarily* and for Python 3 only libraries and applications to emerge. Whilst there are workarounds we *are* in a situation that Python 2 and Python 3 share environment variables for the location of libraries and executing code on startup, whilst at the same time they are largely incompatible and need separate library paths and startup code. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ralf at brainbot.com Thu Jan 14 22:25:31 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Thu, 14 Jan 2010 22:25:31 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <4B4F6E30.50400@egenix.com> (M.'s message of "Thu, 14 Jan 2010 20:19:12 +0100") References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> Message-ID: <871vhswf90.fsf@brainbot.com> "M.-A. Lemburg" writes: > > Naive users won't have PYTHONPATH or any other Python environment > variables setup. > > Really, if you know that you are going to run Python 3 instead of > Python 2 or vice-versa it's easy enough to run You don't even know that you're going to run python. I have 40 python scripts in my /usr/bin directory. > > . py3env.sh > or > . py2env.sh > > in order to setup your shell environment, much like you typically > do when using virtual Python installations or work on different > projects that require different setups. No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > > If you just want to separate Python 2 and 3 files, use the > user site-packages dir which includes the Python version. Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to install those programs in a user's home directory. There are still people running python <2.6. From mal at egenix.com Thu Jan 14 22:51:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Jan 2010 22:51:07 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <871vhswf90.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <0E5A3C2E-1134-40A3-8735-C8C72A685811@gmail.com> <4B4E3E8D.2010407@gmail.com> <4B4F6E30.50400@egenix.com> <871vhswf90.fsf@brainbot.com> Message-ID: <4B4F91CB.2060106@egenix.com> Ralf Schmitt wrote: > "M.-A. Lemburg" writes: > >> >> Naive users won't have PYTHONPATH or any other Python environment >> variables setup. >> >> Really, if you know that you are going to run Python 3 instead of >> Python 2 or vice-versa it's easy enough to run > > You don't even know that you're going to run python. I have 40 python > scripts in my /usr/bin directory. > >> >> . py3env.sh >> or >> . py2env.sh >> >> in order to setup your shell environment, much like you typically >> do when using virtual Python installations or work on different >> projects that require different setups. > > No, thanks. I'd rather setup my shell environment in ~/.zshrc, once. > >> >> If you just want to separate Python 2 and 3 files, use the >> user site-packages dir which includes the Python version. > > Both the bzr and mercurial wiki recommend setting PYTHONPATH in order to > install those programs in a user's home directory. There are still > people running python <2.6. Note that you only need to have a different PYTHONPATH setups for Python 2.x/3.x if you plan to run code that: * is not installed in site-packages (most OS shipped code will be found in the resp. system site-packages/ dir) * is not installed in a user site-packages directory (user installed code will typically go there (*)) * uses modules/packages that come in two different versions (one for Python 2.x and one for 3.x) and use the same name for both versions * needs to work in both Python 2.x and 3.x (*) The method for installing code in user site-packages dir is running: python setup.py install --user instead of the standard python setup.py install which install to the system-wide site-packages. BTW: I guess the bzr and mercurial wikis will need to be updated accordingly - at least for users of Python >=2.6. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Jan 14 22:55:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jan 2010 22:55:04 +0100 Subject: [Python-Dev] [ANN] Python 2.5.5 Release Candidate 1. Message-ID: <4B4F92B8.30806@v.loewis.de> On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.5. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.6 (which is 2.6.4 at this point). This releases fixes issues with the logging and tarfile modules, and with thread-local variables. See the detailed release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed. For more information on Python 2.5.5, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.5.5 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin at v.loewis.de Python Release Manager (on behalf of the entire python-dev team) From status at bugs.python.org Fri Jan 15 18:07:26 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Jan 2010 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100115170726.9963A7865F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (01/08/10 - 01/15/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2536 open (+32) / 16993 closed (+16) / 19529 total (+48) Open issues with patches: 1024 Average duration of open issues: 710 days. Median duration of open issues: 469 days. Open Issues Breakdown open 2501 (+32) pending 34 ( +0) Issues Created Or Reopened (53) _______________________________ PYTHON3PATH environment variable to supersede PYTHONPATH for mul 01/13/10 CLOSED http://bugs.python.org/issue2375 reopened r.david.murray Mark the compiler package as deprecated 01/13/10 http://bugs.python.org/issue6837 reopened ezio.melotti test_distutils failure 01/15/10 CLOSED http://bugs.python.org/issue6961 reopened flox buildbot test_urllib: unsetting missing 'env' variable 01/08/10 CLOSED http://bugs.python.org/issue7026 reopened flox Two float('nan') are not equal 01/08/10 CLOSED http://bugs.python.org/issue7660 created jmfauth compiling ctypes fails with non-ascii path 01/08/10 http://bugs.python.org/issue7661 created pitrou patch time.utcoffset() 01/09/10 http://bugs.python.org/issue7662 created techtonik UCS4 build incorrectly translates cases for non-BMP code points 01/10/10 CLOSED http://bugs.python.org/issue7663 created exarkun python logger does not handle IOError Exception 01/10/10 CLOSED http://bugs.python.org/issue7664 created yateenjoshi test_urllib2 and test_ntpath fail if path contains "\" 01/10/10 http://bugs.python.org/issue7665 created flox test_lib2to3 fails if path contains space 01/10/10 CLOSED http://bugs.python.org/issue7666 created flox test_doctest fails with non-ascii path 01/10/10 http://bugs.python.org/issue7667 created flox buildbot test_httpservers fails with non-ascii path 01/10/10 http://bugs.python.org/issue7668 created flox buildbot test_unicode_file fails with non-ascii path 01/10/10 CLOSED http://bugs.python.org/issue7669 created flox _sqlite3: Block *all* operations on a closed Connection object 01/10/10 http://bugs.python.org/issue7670 created haypo patch test_popen fails if path contains special char like ";" 01/11/10 http://bugs.python.org/issue7671 reopened flox patch _ssl module causes segfault 01/10/10 http://bugs.python.org/issue7672 created ssoria patch audioop: check that length is a multiple of the size 01/11/10 http://bugs.python.org/issue7673 created haypo patch select.select() corner cases: duplicate fds, out-of-range fds 01/11/10 http://bugs.python.org/issue7674 created cdleary help() doesn't accept unicode literals in built in docstrings 01/11/10 http://bugs.python.org/issue7675 created psd patch IDLE shell shouldn't use TABs 01/11/10 http://bugs.python.org/issue7676 created cben distutils, better error message for setup.py upload -sign withou 01/11/10 http://bugs.python.org/issue7677 created illume subprocess.Popen pipeline example code in the documentation is l 01/11/10 http://bugs.python.org/issue7678 created steven.k.wong Warning building 2.7 on OS X 10.6 libintl.h "Present But Cannot 01/12/10 http://bugs.python.org/issue7679 created ssteinerX pythonw crash while attempting to start() a thread object 01/12/10 http://bugs.python.org/issue7680 created dontbugme Incorrect division in [wave.py] 01/12/10 CLOSED http://bugs.python.org/issue7681 created alfps patch, needs review Optimisation of if with constant expression 01/12/10 http://bugs.python.org/issue7682 created sdefresne patch Wrong link in HTML documentation 01/12/10 CLOSED http://bugs.python.org/issue7683 created francescor decimal.py: infinity coefficients in tuples 01/12/10 http://bugs.python.org/issue7684 created skrah minor typo in re docs 01/12/10 CLOSED http://bugs.python.org/issue7685 created mikejs patch redundant open modes 'rbb', 'wbb', 'abb' no longer work on Windo 01/13/10 http://bugs.python.org/issue7686 created ivank Bluetooth support untested 01/13/10 http://bugs.python.org/issue7687 created pitrou TypeError: __name__ must be set to a string object 01/13/10 http://bugs.python.org/issue7688 created frankmillman Pickling of classes with a metaclass and copy_reg 01/13/10 http://bugs.python.org/issue7689 created craigcitro patch Wrong PEP number in importlib module manual page 01/13/10 CLOSED http://bugs.python.org/issue7690 created francescor test_bsddb3 files are not always removed when test fails 01/13/10 CLOSED http://bugs.python.org/issue7691 created flox buildbot Pointless comparision of unsigned integer with zero 01/13/10 CLOSED http://bugs.python.org/issue7692 created Bugger tarfile.extractall can't have unicode extraction path 01/13/10 http://bugs.python.org/issue7693 created pbienst DeprecationWarning from distuils.commands.build_ext needs stackl 01/13/10 http://bugs.python.org/issue7694 created tseaver patch missing termios constants 01/13/10 http://bugs.python.org/issue7695 created conorh Improve Memoryview/Buffer documentation 01/13/10 http://bugs.python.org/issue7696 created flox os.getcwd() should raise UnicodeDecodeError for arbitrary bytes 01/13/10 CLOSED http://bugs.python.org/issue7697 created flox pystack macro in Misc/gdbinit incorrectly uses PyEval_EvalFrame 01/14/10 CLOSED http://bugs.python.org/issue7698 created exarkun patch strptime, strftime documentation 01/14/10 http://bugs.python.org/issue7699 created mikejs patch "-3" flag does not work anymore 01/14/10 CLOSED http://bugs.python.org/issue7700 created flox patch fix output string length for binascii.b2a_uu() 01/14/10 CLOSED http://bugs.python.org/issue7701 created haypo patch Wrong order of parameters of _get_socket in SMTP class in smtpli 01/14/10 http://bugs.python.org/issue7702 created alito Replace buffer()-->memoryview() in Lib/ctypes/test/ 01/14/10 http://bugs.python.org/issue7703 created flox patch Math calculation problem (1.6-1.0)>0.6, python said TRUE 01/14/10 CLOSED http://bugs.python.org/issue7704 created DhaReaL libpython2.6.so is not linked correctly on FreeBSD when threads 01/15/10 http://bugs.python.org/issue7705 created aballier patch, needs review Missing #include guards 01/15/10 http://bugs.python.org/issue7706 created akrpic77 patch multiprocess.Queue operations during import can lead to deadlock 01/15/10 http://bugs.python.org/issue7707 created kripken Conversion of longs to bytes and vice-versa. 01/11/10 http://bugs.python.org/issue1023290 reopened mark.dickinson patch Issues Now Closed (67) ______________________ segfault in curses when calling redrawwin() before refresh() 825 days http://bugs.python.org/issue1266 mark.dickinson "RuntimeError: dictionary changed size during iteration" in Tkin 751 days http://bugs.python.org/issue1658 flox patch Exception exceptions.AttributeError '_shutdown' in 0.6, python said TRUE 0 days http://bugs.python.org/issue7704 tim_one Support for sending multipart form data 2452 days http://bugs.python.org/issue727898 r.david.murray email.MIME*.as_string removes duplicate spaces 1440 days http://bugs.python.org/issue1422094 r.david.murray easy test_parsedate_acceptable_to_time_functions+DST == :-( 1394 days http://bugs.python.org/issue1454285 r.david.murray email module does not complay with RFC 2046: CRLF issue 1196 days http://bugs.python.org/issue1571841 r.david.murray email.FeedParser.BufferedSubFile improperly handles "\r\n" 969 days http://bugs.python.org/issue1721862 r.david.murray patch, easy Top Issues Most Discussed (10) ______________________________ 20 dtoa.c: oversize b in quorem 11 days open http://bugs.python.org/issue7632 18 _ssl module causes segfault 5 days open http://bugs.python.org/issue7672 12 Speed up cPickle's pickling generally 287 days open http://bugs.python.org/issue5683 8 OS X pythonw.c compile error with 10.4 or earlier deployment ta 7 days open http://bugs.python.org/issue7658 8 Fatal error on thread creation in low memory condition 27 days open http://bugs.python.org/issue7544 8 Direct calls to PyObject_Del/PyObject_DEL are broken for --with 558 days open http://bugs.python.org/issue3299 7 "-3" flag does not work anymore 1 days closed http://bugs.python.org/issue7700 7 tarfile.extractall can't have unicode extraction path 2 days open http://bugs.python.org/issue7693 7 test_urllib: unsetting missing 'env' variable 0 days closed http://bugs.python.org/issue7026 7 os.path.abspath with unicode argument should call os.getcwdu 542 days open http://bugs.python.org/issue3426 From g.brandl at gmx.net Sat Jan 16 21:05:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 16 Jan 2010 21:05:46 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <20100113200840.GC14858@phd.pp.ru> <319e029f1001131227jf4006d3ya562fee26d22f3dd@mail.gmail.com> Message-ID: Am 13.01.2010 21:27, schrieb Lennart Regebro: > On Wed, Jan 13, 2010 at 21:08, Oleg Broytman wrote: >> On Wed, Jan 13, 2010 at 08:50:59PM +0100, Lennart Regebro wrote: >>> What do you need to do in the PYTHONSTARTUP file? >>> Ten years of Python programming, and I didn't even know it existed. :-) >> >> See http://phd.pp.ru/Software/dotfiles/init.py.html for an example. > > Cheese and fries! :-) > > Anyway, I don't see how this could pose any problems, because any user > advanced enough to do these things will be advanced enough to > understand what the problem is and fix it so it works under Python 3 > too. I'd propose adding a bit of text to the environment variables documentation, and especially about the needs of PYTHONSTARTUP files. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From asmodai at in-nomine.org Sat Jan 16 21:50:56 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 16 Jan 2010 21:50:56 +0100 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <874ompg225.fsf@brainbot.com> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> Message-ID: <20100116205056.GL99670@nexus.in-nomine.org> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >hehe. tab completion: With bpython and ipython available, why would you even want to stick to the 'plain old' interactive interpreter? (Sorry to derail the discussion, but maybe there's more people that have not heard of either or both.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B For ever, brother, hail and farewell... From solipsis at pitrou.net Sat Jan 16 21:57:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Jan 2010 20:57:13 +0000 (UTC) Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: Jeroen Ruigrok van der Werven in-nomine.org> writes: > > -On [20100113 22:13], Ralf Schmitt (ralf brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick to the > 'plain old' interactive interpreter? Why wouldn't we? There are probably many more people using the standard Python prompt than ipython. From ben+python at benfinney.id.au Sat Jan 16 23:41:03 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Jan 2010 09:41:03 +1100 Subject: [Python-Dev] PYTHON3PATH References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> Message-ID: <87y6jxk70g.fsf@benfinney.id.au> Jeroen Ruigrok van der Werven writes: > -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: > >hehe. tab completion: > > With bpython and ipython available, why would you even want to stick > to the 'plain old' interactive interpreter? Because those optional extras are not always available, whereas the standard interactive interpreter is always available with CPython distributions. -- \ ?I fly Air Bizarre. You buy a combination one-way round-trip | `\ ticket. Leave any Monday, and they bring you back the previous | _o__) Friday. That way you still have the weekend.? ?Steven Wright | Ben Finney From ncoghlan at gmail.com Sun Jan 17 01:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:22:10 +1000 Subject: [Python-Dev] PYTHON3PATH In-Reply-To: <87y6jxk70g.fsf@benfinney.id.au> References: <20100113172442.4A7771FA679@kimball.webabinitio.net> <87r5ptdhu3.fsf@muni.brainbot.com> <319e029f1001131150o7143f9bhacc9eb256ca30f76@mail.gmail.com> <874ompg225.fsf@brainbot.com> <20100116205056.GL99670@nexus.in-nomine.org> <87y6jxk70g.fsf@benfinney.id.au> Message-ID: <4B525832.8090904@gmail.com> Ben Finney wrote: > Jeroen Ruigrok van der Werven writes: > >> -On [20100113 22:13], Ralf Schmitt (ralf at brainbot.com) wrote: >>> hehe. tab completion: >> With bpython and ipython available, why would you even want to stick >> to the 'plain old' interactive interpreter? > > Because those optional extras are not always available, whereas the > standard interactive interpreter is always available with CPython > distributions. I've never even contemplated the idea of installing 3rd party apps for the 4 current development builds (2.x trunk, 2.x maintenance, 3.x trunk, 3.x maintenance) on my home development machine. And of course work suffers from the problem of not being allowed to install arbitrary apps I downloaded from the internet without getting the licensing vetted for commercial acceptability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 01:34:08 2010 From: nad at acm.org (Ned Deily) Date: Sat, 16 Jan 2010 16:34:08 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? Message-ID: I've recently seen a couple of references to 3.1.2 go by in checkins which made me wonder whether dates have been proposed yet for updates to either 3.1 or 2.6. I don't recall seeing any and I didn't see any references in the PEPs. Some advance warning would be nice. I assume that some critical problem could trigger planning for an update on short notice; is there a time-limit trigger as well? -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Jan 17 01:55:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 10:55:38 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52600A.7060201@gmail.com> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Usually there is one more regular maintenance release of the existing maintenance branches within a few months of the creation of the next version (releases before then are at the discretion of the release manager, and security releases continue for much longer). So take the planned 2.7 and 3.2 release dates and add a couple of months to each one to get the likely release dates for the 2.6.x and 3.1.x maintenance releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sun Jan 17 10:21:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 17 Jan 2010 10:21:49 +0100 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B52D6AD.6090302@v.loewis.de> Ned Deily wrote: > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. I assume > that some critical problem could trigger planning for an update on short > notice; is there a time-limit trigger as well? Barry was once proposing time-based releases; this idea didn't really catch on. It's the release manager who decides when the next bug fix release will be made, often in response to developers asking for one. Regards, Martin From solipsis at pitrou.net Sun Jan 17 14:00:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 13:00:04 +0000 (UTC) Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: Message-ID: Ned Deily acm.org> writes: > > I've recently seen a couple of references to 3.1.2 go by in checkins > which made me wonder whether dates have been proposed yet for updates to > either 3.1 or 2.6. I don't recall seeing any and I didn't see any > references in the PEPs. Some advance warning would be nice. There are a couple of release blockers right now. Once they are fixed or deferred, I think it would be nice to have a 3.1.2. Why do you need "some advance warning" though? From ncoghlan at gmail.com Sun Jan 17 14:40:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Jan 2010 23:40:42 +1000 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: References: Message-ID: <4B53135A.7060104@gmail.com> Antoine Pitrou wrote: > Ned Deily acm.org> writes: >> I've recently seen a couple of references to 3.1.2 go by in >> checkins which made me wonder whether dates have been proposed yet >> for updates to either 3.1 or 2.6. I don't recall seeing any and I >> didn't see any references in the PEPs. Some advance warning would >> be nice. > > There are a couple of release blockers right now. Once they are fixed > or deferred, I think it would be nice to have a 3.1.2. Why do you > need "some advance warning" though? Advance warning does allow interested users that would consider upgrading to schedule time for testing before the maintenance release comes out. This is particularly useful in helping to make a 1-week RC period effective in picking up issues that might otherwise lead to a brown paper bag release to fix major issues that slipped through our own automated test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From nad at acm.org Sun Jan 17 19:01:37 2010 From: nad at acm.org (Ned Deily) Date: Sun, 17 Jan 2010 10:01:37 -0800 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? References: <4B53135A.7060104@gmail.com> Message-ID: In article <4B53135A.7060104 at gmail.com>, Nick Coghlan wrote: > Antoine Pitrou wrote: > > Ned Deily acm.org> writes: > >> I've recently seen a couple of references to 3.1.2 go by in > >> checkins which made me wonder whether dates have been proposed yet > >> for updates to either 3.1 or 2.6. I don't recall seeing any and I > >> didn't see any references in the PEPs. Some advance warning would > >> be nice. > > There are a couple of release blockers right now. Once they are fixed > > or deferred, I think it would be nice to have a 3.1.2. Why do you > > need "some advance warning" though? > > Advance warning does allow interested users that would consider > upgrading to schedule time for testing before the maintenance release > comes out. This is particularly useful in helping to make a 1-week RC > period effective in picking up issues that might otherwise lead to a > brown paper bag release to fix major issues that slipped through our own > automated test coverage. That. and resource contention: there are always potential fixes in the pipeline that could or should be bumped in priority if one knows there is a code cutoff approaching. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Sun Jan 17 20:51:58 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 20:51:58 +0100 Subject: [Python-Dev] Enhancing the shutil module Message-ID: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Hello, For 2.7/3.2, I am in the process of removing modules in Distutils that can be replaced by calls to existing functions in stdlib. For instance, "dir_util" and "file_util" (old modules from the Python 1.x era) are going away in favor of calls to shutil (and os), so the Distutils package gets lighter. Another module I would like to move away from Distutils is "archive_util". It contains helpers to build archives, whether they are zip or tar files. I propose to move those useful functions into shutil, as this seems the most logical place. I also propose to maintain this "shutil" module for now on (no one is declared as a maintainer in maintainers.rst) since Distutils will become a heavy user of its functions. Any objections/opinions ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 20:53:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 19:53:03 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536A9F.5050203@voidspace.org.uk> On 17/01/2010 19:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > > Regards, > Tarek > > I think it's a great idea. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Sun Jan 17 20:55:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2010 11:55:47 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > If it's archive-agnostic then shutil is probably the best place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Great! > Any objections/opinions ? > No objections from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jan 17 21:04:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Jan 2010 20:04:41 +0000 (UTC) Subject: [Python-Dev] Enhancing the shutil module References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: Tarek Ziad? gmail.com> writes: > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. Are these functions portable? Do they rely on external programs? > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. It's ok for me. Regards Antoine. From ziade.tarek at gmail.com Sun Jan 17 21:09:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:09:18 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: > > > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >> >> Hello, >> >> For 2.7/3.2, I am in the process of removing modules in Distutils that >> can be replaced by calls to existing functions in stdlib. For >> instance, "dir_util" and "file_util" (old modules from the Python 1.x >> era) are going away in favor of calls to shutil (and os), so the >> Distutils package gets lighter. >> >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. > > If it's archive-agnostic then shutil is probably the best place. In more details: It allows the creation of gzip, bzip2, tar and zip files through a single API. There's a registry of supported formats and the API is driven by a format identifier. To do the work it uses stdlib's compression modules. Although it tries the "zip" system command as a fallback if the "zipfile" module is not present. (notice that I've removed the support of "compress" (.Z) some time ago) Regards Tarek -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Sun Jan 17 21:15:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 17 Jan 2010 20:15:00 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <4B536FC4.9090304@voidspace.org.uk> On 17/01/2010 20:04, Antoine Pitrou wrote: > Tarek Ziad? gmail.com> writes: > >> Another module I would like to move away from Distutils is >> "archive_util". It contains helpers to build archives, whether they >> are zip or tar files. I propose to move those useful functions into >> shutil, as this seems the most logical place. >> > Are these functions portable? Do they rely on external programs? > > I believe that part of the work that Tarek has been doing has been to make these distutils commands use the Python standard library and not depend on external programs. In which case they seem like *excellent* additions to the shutil module. Of course Tarek can speak for himself... Michael >> I also propose to maintain this "shutil" module for now on (no one is >> declared as a maintainer in maintainers.rst) since Distutils will >> become a heavy user of its functions. >> > It's ok for me. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Sun Jan 17 21:43:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 21:43:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B536FC4.9090304@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> Message-ID: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> On Sun, Jan 17, 2010 at 9:15 PM, Michael Foord wrote: [..] >> Are these functions portable? Do they rely on external programs? >> >> > > I believe that part of the work that Tarek has been doing has been to make > these distutils commands use the Python standard library and not depend on > external programs. In which case they seem like *excellent* additions to the > shutil module. yes, in the past the "tar" files where created using the "tar" command but this has been changed. For a while now, they are portable and they use stdlib code only. A recent addition is to be able to define user/group permissions in the tar files, thanks to Lars' work in the tarfile module. There's one remaining external call for "zip" done if the zip module is not found, but I am happy to remove it and throw an exception if it's not found, and keep the external "zip" call on Distutils side, so shutil stays 100% stdlib-powered. > Of course Tarek can speak for himself... Thanks for explaining it ! :) Regards Tarek -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Sun Jan 17 22:50:52 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Sun, 17 Jan 2010 13:50:52 -0800 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> Message-ID: <4B53863C.5060304@activestate.com> On 1/17/2010 12:09 PM, Tarek Ziad? wrote: > On Sun, Jan 17, 2010 at 8:55 PM, Brett Cannon wrote: >> > On Sun, Jan 17, 2010 at 11:51, Tarek Ziad? wrote: >>> >> Another module I would like to move away from Distutils is >>> >> "archive_util". It contains helpers to build archives, whether they >>> >> are zip or tar files. I propose to move those useful functions into >>> >> shutil, as this seems the most logical place. >> > If it's archive-agnostic then shutil is probably the best place. > In more details: > It allows the creation of gzip, bzip2, tar and zip files through a single API. > There's a registry of supported formats and the API is driven by a > format identifier. Will it also allow decompression of the said archive types? Distribute has some utility code to handle zip/tar archives. So does PyPM. This is because the `tarfile` and `zipfile` modules do not "just work" due to several issues. See http://gist.github.com/279606 Take note of the following in the above code: 1) _ensure_read_write_access 2) *File.is_valid 3) ZippedFile.extract ... issue 6510 4) ZippedFile.extract ... issue 6609 5) TarredFile.extract ... issue 6584 6) The way unpack() detects the unpacked directory. -srid From ziade.tarek at gmail.com Sun Jan 17 23:09:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 17 Jan 2010 23:09:29 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B53863C.5060304@activestate.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <94bdd2611001171209l4a841dd2ne0848b9501d76470@mail.gmail.com> <4B53863C.5060304@activestate.com> Message-ID: <94bdd2611001171409j4ec1e0bfj47e59894fdec0d8a@mail.gmail.com> On Sun, Jan 17, 2010 at 10:50 PM, Sridhar Ratnakumar wrote: [..] > Will it also allow decompression of the said archive types? No but it would make sense having this one as well. Distribute/Setuptools' "unpack_archive" (used by easy_install) was implemented using the same principle as Distutils' "make_archive". I can add it as well while I am at it : it has been successfully used for years by easy_install. > Distribute has > some utility code to handle zip/tar archives. So does PyPM. This is because > the `tarfile` and `zipfile` modules do not "just work" due to several > issues. > > See http://gist.github.com/279606 > > Take note of the following in the above code: > > ?1) _ensure_read_write_access > ?2) *File.is_valid > ?3) ZippedFile.extract ... issue 6510 > ?4) ZippedFile.extract ... issue 6609 > ?5) TarredFile.extract ... issue 6584 Looks like some of these are already fixed (I just looked quickly in the bug tracker though) If its not already done and pending, it would be great if you could refactor your fixes into patches for the remaining issues for each one of those modules Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Sun Jan 17 23:56:37 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 17 Jan 2010 17:56:37 -0500 Subject: [Python-Dev] 3.1.2 / 2.6.5 maintenance releases? In-Reply-To: <4B52D6AD.6090302@v.loewis.de> References: <4B52D6AD.6090302@v.loewis.de> Message-ID: On Jan 17, 2010, at 4:21 AM, Martin v. L?wis wrote: > Barry was once proposing time-based releases; this idea didn't really > catch on. I'm still a proponent of this, but it doesn't seem like there's enough support for it. > It's the release manager who decides when the next bug fix release will > be made, often in response to developers asking for one. I'm happy to make a 2.6.5 release whenever it makes sense. -Barry From ncoghlan at gmail.com Mon Jan 18 13:40:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Jan 2010 22:40:01 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> Message-ID: <4B5456A1.2080109@gmail.com> Tarek Ziad? wrote: > There's one remaining external call for "zip" done if the zip module > is not found, but I am happy to remove it and throw an exception if > it's not found, and keep the external "zip" call on Distutils side, so > shutil stays 100% stdlib-powered. +1 for that approach. These changes all sound like nice additions to shutil, and hooray for every module that gets adopted by an active maintainer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From masklinn at masklinn.net Mon Jan 18 14:34:14 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 14:34:14 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5456A1.2080109@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> Message-ID: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: > > Tarek Ziad? wrote: >> There's one remaining external call for "zip" done if the zip module >> is not found, but I am happy to remove it and throw an exception if >> it's not found, and keep the external "zip" call on Distutils side, so >> shutil stays 100% stdlib-powered. > > +1 for that approach. These changes all sound like nice additions to > shutil, and hooray for every module that gets adopted by an active > maintainer :) Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. From doug.hellmann at gmail.com Mon Jan 18 14:46:05 2010 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 18 Jan 2010 08:46:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils >>> side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil > advertises itself as "a number of high-level operations on files and > collections of files." and from what I understood it was a bunch of > shell-type utility functions to easily copy, move or remove files > and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/ > objects in a new module separate from shutil, dealing specifically > with cross-archive APIs and linked from the current archive-specific > modules (essentially, just take the current archive_util, move it to > the toplevel of the stdlib and maybe rename it)? It would also make > the module much easier to find when searching through the module > listing, I think. +1 From fuzzyman at voidspace.org.uk Mon Jan 18 14:57:49 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 18 Jan 2010 13:57:49 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> Message-ID: <4B5468DD.5040200@voidspace.org.uk> On 18/01/2010 13:46, Doug Hellmann wrote: > > On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > >> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>> >>> Tarek Ziad? wrote: >>>> There's one remaining external call for "zip" done if the zip module >>>> is not found, but I am happy to remove it and throw an exception if >>>> it's not found, and keep the external "zip" call on Distutils side, so >>>> shutil stays 100% stdlib-powered. >>> >>> +1 for that approach. These changes all sound like nice additions to >>> shutil, and hooray for every module that gets adopted by an active >>> maintainer :) >> >> Isn't it a bit weird to include that to shutil though? shutil >> advertises itself as "a number of high-level operations on files and >> collections of files." Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> and from what I understood it was a bunch of shell-type utility functions like tar and zip? :-) >> to easily copy, move or remove files and directories (that's pretty >> much all there is in it at this time). >> I don't think the additions are out of place prima-facie. >> Wouldn't it make more sense to put those "archive utils" >> functions/objects in a new module separate from shutil, dealing >> specifically with cross-archive APIs and linked from the current >> archive-specific modules (essentially, just take the current >> archive_util, move it to the toplevel of the stdlib and maybe rename >> it)? It would also make the module much easier to find when searching >> through the module listing, I think. > > +1 > Proliferation of modules is itself a bad thing though. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Mon Jan 18 15:34:12 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 15:34:12 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <94bdd2611001180634sacd998fm6f67dc680e2e61c3@mail.gmail.com> On Mon, Jan 18, 2010 at 2:57 PM, Michael Foord wrote: [..] >>> Wouldn't it make more sense to put those "archive utils" >>> functions/objects in a new module separate from shutil, dealing specifically >>> with cross-archive APIs and linked from the current archive-specific modules >>> (essentially, just take the current archive_util, move it to the toplevel of >>> the stdlib and maybe rename it)? It would also make the module much easier >>> to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. I am with Michael here. I think having this function in shutil is the right place. For the find problem, I think docs.python.org is easy to search now, as long as the shutil module documentation has an good set of examples for this new API. We could even add references in the tarfile, zipfile modules documentation pointing to these examples. Regards Tarek -- Tarek Ziad? | http://ziade.org From masklinn at masklinn.net Mon Jan 18 15:56:05 2010 From: masklinn at masklinn.net (Masklinn) Date: Mon, 18 Jan 2010 15:56:05 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <4B5468DD.5040200@voidspace.org.uk> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> Message-ID: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> On 18 Jan 2010, at 14:57 , Michael Foord wrote: > > On 18/01/2010 13:46, Doug Hellmann wrote: >> >> On Jan 18, 2010, at 8:34 AM, Masklinn wrote: >> >>> On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >>>> >>>> Tarek Ziad? wrote: >>>>> There's one remaining external call for "zip" done if the zip module >>>>> is not found, but I am happy to remove it and throw an exception if >>>>> it's not found, and keep the external "zip" call on Distutils side, so >>>>> shutil stays 100% stdlib-powered. >>>> >>>> +1 for that approach. These changes all sound like nice additions to >>>> shutil, and hooray for every module that gets adopted by an active >>>> maintainer :) >>> >>> Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." > > Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? > Well no those are high-level operations on a very restricted set of file types (archives). >>> and from what I understood it was a bunch of shell-type utility functions > > like tar and zip? :-) > Tar and zip have a module each at this time, so they're considered pretty big. I don't think anybody would consider "mv" big enough to give it a module on its own. >>> Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. >> >> +1 >> > > Proliferation of modules is itself a bad thing though. Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. From phd at phd.pp.ru Mon Jan 18 16:03:38 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 18 Jan 2010 18:03:38 +0300 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118150337.GA19391@phd.pp.ru> On Mon, Jan 18, 2010 at 02:34:14PM +0100, Masklinn wrote: > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. +1 Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ziade.tarek at gmail.com Mon Jan 18 16:24:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 18 Jan 2010 16:24:37 +0100 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <2D626E52-E592-44C8-A8E8-C72FABE6AFEA@gmail.com> <4B5468DD.5040200@voidspace.org.uk> <6A054C33-75FB-48E6-9AD1-FE6F0ADF5940@masklinn.net> Message-ID: <94bdd2611001180724u7f48e620ub32e13ef96c873b9@mail.gmail.com> On Mon, Jan 18, 2010 at 3:56 PM, Masklinn wrote: [..] >> Well - isn't what's being proposed "a number of high-level operations on files and collections of files." ? >> > Well no those are high-level operations on a very restricted set of file types (archives) not really: make_archive/unpack_archive, are also dealing with files and directories. [..] >> Proliferation of modules is itself a bad thing though. > Yes, but so is the loss of focus of a module. I hate using that argument, but I fear this would be a step on a slippery slope of making shutil a monster-bag of anything that has to do with inodes, no matter how tenuous or removed the connection is. > > Plus having this as a toplevel module/package would open the window to moving all archive-related modules within it (in the py4 window), ? la xml package without having to move itself. I am not sure why this would happen. For instance, shutil is already on the top of the os module since a few major versions IIRC, because it reads and writes files and directories. But it was not moved into the os package (or vice-versa) The shutil module uses APIs to read and write files. So if it works with archives, it's just a specific read/write API that is used, but that doesn't mean tarfile and zipfile might be reunited with shutil imho. If the shutil module is restricted to high-level files and directories manipulation, working with archives is just a target like another I think. But at the end I am 0- to create a new module, because what really matters to me is to take it out of Distutils :) Regards, Tarek -- Tarek Ziad? | http://ziade.org From listsin at integrateddevcorp.com Mon Jan 18 16:56:05 2010 From: listsin at integrateddevcorp.com (Steve Steiner (listsin)) Date: Mon, 18 Jan 2010 10:56:05 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: On Jan 18, 2010, at 8:34 AM, Masklinn wrote: > On 18 Jan 2010, at 13:40 , Nick Coghlan wrote: >> >> Tarek Ziad? wrote: >>> There's one remaining external call for "zip" done if the zip module >>> is not found, but I am happy to remove it and throw an exception if >>> it's not found, and keep the external "zip" call on Distutils side, so >>> shutil stays 100% stdlib-powered. >> >> +1 for that approach. These changes all sound like nice additions to >> shutil, and hooray for every module that gets adopted by an active >> maintainer :) > > Isn't it a bit weird to include that to shutil though? shutil advertises itself as "a number of high-level operations on files and collections of files." and from what I understood it was a bunch of shell-type utility functions to easily copy, move or remove files and directories (that's pretty much all there is in it at this time). > > Wouldn't it make more sense to put those "archive utils" functions/objects in a new module separate from shutil, dealing specifically with cross-archive APIs and linked from the current archive-specific modules (essentially, just take the current archive_util, move it to the toplevel of the stdlib and maybe rename it)? It would also make the module much easier to find when searching through the module listing, I think. As much of a pain as it is to get new modules accepted, I agree that mixing archiving functions into shutil is not the right way to do it and that a separate archive_util module would make much more sense and would give a logical place to put any extensions to archive handling. S From rdmurray at bitdance.com Mon Jan 18 20:28:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 18 Jan 2010 14:28:47 -0500 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> Message-ID: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: > As much of a pain as it is to get new modules accepted, I agree that > mixing archiving functions into shutil is not the right way to do it > and that a separate archive_util module would make much more sense and > would give a logical place to put any extensions to archive handling. Looking at the source code and API for both shutil and archive_util, I think that the archive_util methods fit into shutil. shutil currently wraps some standard library facilities with convenience functions for operations you might otherwise perform at the shell command line using OS facilities. As far as I can tell, archive_util does the same, and seems quite within the shutil mission of "high level file operations". So +1 from me for putting these in shutil. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From p.f.moore at gmail.com Mon Jan 18 20:48:43 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 18 Jan 2010 19:48:43 +0000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> Message-ID: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> 2010/1/18 R. David Murray : > On Mon, 18 Jan 2010 10:56:05 -0500, "Steve Steiner (listsin)" wrote: >> As much of a pain as it is to get new modules accepted, I agree that >> mixing archiving functions into shutil is not the right way to do it >> and that a separate archive_util module would make much more sense and >> would give a logical place to put any extensions to archive handling. > > Looking at the source code and API for both shutil and archive_util, I > think that the archive_util methods fit into shutil. ?shutil currently > wraps some standard library facilities with convenience functions > for operations you might otherwise perform at the shell command line using > OS facilities. ?As far as I can tell, archive_util does the same, and > seems quite within the shutil mission of "high level file operations". > > So +1 from me for putting these in shutil. Conceptually, I'm happy with these going into shutil (and +1 on the rest of Tarek's proposal, too!) To my mind, shutil is a module for higher-level operations on files - the sort of things you'd do in shell commands, like move a batch of files around (mv), create a directory tree (mkdir -p). Tarring or zipping up a batch of files fits nicely into that space. Paul. From david.lyon at preisshare.net Tue Jan 19 02:53:43 2010 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 18 Jan 2010 20:53:43 -0500 Subject: [Python-Dev] PyCon Keynote In-Reply-To: References: Message-ID: <0dfb370163cf3a284ca256f49662db74@preisshare.net> On Wed, 13 Jan 2010 10:51:14 -0800, Guido van Rossum wrote: > Please mail me topics you'd like to hear me talk about in my keynote > at PyCon this year. As a typical (but perphaps more vocal) python developer I would like to hear things that might aspire me and others to move to python 3. The way I see it is that python 2.x represents everyones comfort zone. It's safe and nobody got fired at work for using python 2.x I would like to hear how python 3 could be 'shaken' up with a slightly less conservative policy that has gone with the python 2.x series. The logic perhaps being that since only a minority use the 3.x series, it's functionality set is still more up in the air. imo it needs bigger batteries to give it more power than the 2.x series. This meaning that the stdlib could take an extra 5-10 packages not in the 2.x series. Just as one example, sphinx - a great documentation tool. I can easily name five or six others. I'd also like to hear more of your ideas on pypi and getting it to have as much who-ha as CPAN. You could do a lot to enlarge the developer group. Help us all get our priorities to be inline with your own wishes and expectations. If you ask us all to put in a big year and buy you a beer at the end.. I'm certain we all will. Every little bit of inspiration and direction counts for a lot... Best Regards David From ncoghlan at gmail.com Tue Jan 19 12:20:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Jan 2010 21:20:05 +1000 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> <4B536FC4.9090304@voidspace.org.uk> <94bdd2611001171243i604b612ct2db654ffe3812ec8@mail.gmail.com> <4B5456A1.2080109@gmail.com> <96E0938E-D1E1-4751-A06B-2B2BD13830BA@masklinn.net> <20100118192847.1C99C1FB6FE@kimball.webabinitio.net> <79990c6b1001181148y7e48d770n546acfe2f6c62367@mail.gmail.com> Message-ID: <4B559565.8090602@gmail.com> Paul Moore wrote: > 2010/1/18 R. David Murray : >> So +1 from me for putting these in shutil. > > Conceptually, I'm happy with these going into shutil (and +1 on the > rest of Tarek's proposal, too!) > > To my mind, shutil is a module for higher-level operations on files - > the sort of things you'd do in shell commands, like move a batch of > files around (mv), create a directory tree (mkdir -p). Tarring or > zipping up a batch of files fits nicely into that space. This is also reflected in the way at least Windows handles archives these days - it took them a couple of iterations to get it right (and resolve some of the performance impacts), but Explorer now does a decent job of integrating archives into the directory tree as "folders that happen to be compressed". Are archives as fundamental as directories and files? No. But in the context of shutil, the fact that their internal structure is largely about directories and files makes them more than just another arbitrary file type. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Jan 19 14:16:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 08:16:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad Message-ID: <20100119081639.5d431ed9@freewill> I've just updated the Launchpad mirrors for the 4 active Python branches, trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar branches on code.python.org but it's probably been 7 months or so since those were regularly updated. Now the Launchpad branches sync against the read-only Subversion branches at http://svn.python.org, so they should remain up-to-date (within the re-sync timeframe of about 4 hours). This means you can once again use Bazaar to get local branches of Python, and you can of course push your own branches to Launchpad. I believe you can even use the bzr-svn plugin to commit changes back to the Subversion master, though I have not yet tried this. To get a local branch, just do any of the following: % bzr branch lp:python (for trunk) % bzr branch lp:python/2.6 % bzr branch lp:python/py3k % bzr branch lp:python/3.1 (It's fairly easy to create new mirrors for other Subversion branches, e.g. Python 2.5; just drop me an email if you want them.) If you're going to create a lot of branches you probably want to put them in a shared repository. E.g. % bzr init-repo pythonbzr % cd pythonbzr % bzr branch lp:python/py3k Bazaar 2.0 or better is recommended. For me, it took about 5m to check the first branch out from Launchpad, and then about 30s or so for each subsequent branch. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From abpillai at gmail.com Tue Jan 19 14:37:18 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Tue, 19 Jan 2010 19:07:18 +0530 Subject: [Python-Dev] Enhancing the shutil module In-Reply-To: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> References: <94bdd2611001171151w21838b09k4620c562058d9ccc@mail.gmail.com> Message-ID: <8548c5f31001190537l2bcab5cfkb6f461910e4abd8b@mail.gmail.com> On Mon, Jan 18, 2010 at 1:21 AM, Tarek Ziad? wrote: > Hello, > > For 2.7/3.2, I am in the process of removing modules in Distutils that > can be replaced by calls to existing functions in stdlib. For > instance, "dir_util" and "file_util" (old modules from the Python 1.x > era) are going away in favor of calls to shutil (and os), so the > Distutils package gets lighter. > > Another module I would like to move away from Distutils is > "archive_util". It contains helpers to build archives, whether they > are zip or tar files. I propose to move those useful functions into > shutil, as this seems the most logical place. > > I also propose to maintain this "shutil" module for now on (no one is > declared as a maintainer in maintainers.rst) since Distutils will > become a heavy user of its functions. > > Any objections/opinions ? > +1 for this. Just make sure that you change the docstring of shutil which now reads as, " shutil - Utility functions for copying files and directory trees." According to this "definition", archives don't fit in there. But the functionality does fit right in, so just need to make sure that it is reflected in the __doc__ . > Regards, > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/abpillai%40gmail.com > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jan 19 16:50:42 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Jan 2010 15:50:42 +0000 (UTC) Subject: [Python-Dev] Mailing List archive corruption? Message-ID: Hi, When I look at the mailing list archive for python-dev, I see some odd stuff at the bottom of the page: http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 Anyone know what's happened? Regards, Vinay Sajip From barry at python.org Tue Jan 19 17:07:48 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:07:48 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: Message-ID: <20100119110748.69bc564a@freewill> On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: >When I look at the mailing list archive for python-dev, I see some odd stuff at >the bottom of the page: > >http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >Anyone know what's happened? WTF? I think the archives were recently regenerated, so there's probably a fubar there. CC'ing the postmasters. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From foom at fuhm.net Tue Jan 19 17:24:57 2010 From: foom at fuhm.net (James Y Knight) Date: Tue, 19 Jan 2010 11:24:57 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119110748.69bc564a@freewill> References: <20100119110748.69bc564a@freewill> Message-ID: On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > >> When I look at the mailing list archive for python-dev, I see some >> odd stuff at >> the bottom of the page: >> >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 >> >> Anyone know what's happened? > > WTF? I think the archives were recently regenerated, so there's > probably a > fubar there. CC'ing the postmasters. That happens if messages had unescaped "From" lines in the middle of them. No doubt, you've now broken every link anyone had ever made into the python-dev archives, because now all the article numbers are different. BTDT...unfortunately... Pipermail really is quite crappy, sigh. Anyhow, when I did that, I went back to a backup to get the original article numbers, and edited the mbox file escaping From lines or adding additional empty messages until the newly regenerated article numbers matched the originals. I'd highly recommend going through that painful process, since I suspect a *lot* of people have links to the python-dev archive. Hope you have a backup (or can find caches on google or archive.org or something). James From barry at python.org Tue Jan 19 17:44:21 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 11:44:21 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119114421.3b96fbd4@freewill> On Jan 19, 2010, at 11:24 AM, James Y Knight wrote: >No doubt, you've now broken every link anyone had ever made into the >python-dev archives, because now all the article numbers are >different. BTDT...unfortunately... Pipermail really is quite crappy, >sigh. I've been trying for 10+ years to get folks interested in helping me fix this (and a few other warts) in Pipermail, to not much success. ;/ >Anyhow, when I did that, I went back to a backup to get the original >article numbers, and edited the mbox file escaping From lines or >adding additional empty messages until the newly regenerated article >numbers matched the originals. I'd highly recommend going through that >painful process, since I suspect a *lot* of people have links to the >python-dev archive. Hope you have a backup (or can find caches on >google or archive.org or something). bin/cleanarch uses a set of heuristics to find unescaped From lines and fix them. It's generally pretty good, but it certain can change message numbers (and sadly, their urls). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rdmurray at bitdance.com Tue Jan 19 18:48:41 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 19 Jan 2010 12:48:41 -0500 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: References: <20100119110748.69bc564a@freewill> Message-ID: <20100119174841.9BC3B16A53@kimball.webabinitio.net> On Tue, 19 Jan 2010 11:24:57 -0500, James Y Knight wrote: > > On Jan 19, 2010, at 11:07 AM, Barry Warsaw wrote: > > > On Jan 19, 2010, at 03:50 PM, Vinay Sajip wrote: > > > >> When I look at the mailing list archive for python-dev, I see some > >> odd stuff at the bottom of the page: > >> > >> http://mail.python.org/pipermail/python-dev/2010-January/thread.html#95232 > >> > >> Anyone know what's happened? > > > > WTF? I think the archives were recently regenerated, so there's > > probably a fubar there. CC'ing the postmasters. > > That happens if messages had unescaped "From" lines in the middle of > them. > > No doubt, you've now broken every link anyone had ever made into the > python-dev archives, because now all the article numbers are > different. BTDT...unfortunately... Pipermail really is quite crappy, > sigh. > > Anyhow, when I did that, I went back to a backup to get the original > article numbers, and edited the mbox file escaping From lines or > adding additional empty messages until the newly regenerated article > numbers matched the originals. I'd highly recommend going through that > painful process, since I suspect a *lot* of people have links to the > python-dev archive. Hope you have a backup (or can find caches on > google or archive.org or something). The Python issue tracker does, for one. -- R. David Murray www.bitdance.com Business Process Automation - Network/Server Management - Routers/Firewalls From ncoghlan at gmail.com Tue Jan 19 22:18:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 07:18:52 +1000 Subject: [Python-Dev] Mailing List archive corruption? In-Reply-To: <20100119174841.9BC3B16A53@kimball.webabinitio.net> References: <20100119110748.69bc564a@freewill> <20100119174841.9BC3B16A53@kimball.webabinitio.net> Message-ID: <4B5621BC.4070608@gmail.com> R. David Murray wrote: > The Python issue tracker does, for one. And all the PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at pythontest.org Wed Jan 20 00:16:44 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 10:16:44 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119081639.5d431ed9@freewill> References: <20100119081639.5d431ed9@freewill> Message-ID: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Hi Barry, That looks very interesting... So does that mean we could update the stdlib for a given python version using this ? David > I've just updated the Launchpad mirrors for the 4 active Python branches, > trunk, py3k, 2.6, and 3.1. These used to mirror the defunct Bazaar > branches > on code.python.org but it's probably been 7 months or so since those were > regularly updated. Now the Launchpad branches sync against the read-only > Subversion branches at http://svn.python.org, so they should remain > up-to-date > (within the re-sync timeframe of about 4 hours). > > This means you can once again use Bazaar to get local branches of Python, > and > you can of course push your own branches to Launchpad. I believe you can > even > use the bzr-svn plugin to commit changes back to the Subversion master, > though > I have not yet tried this. > > To get a local branch, just do any of the following: > > % bzr branch lp:python (for trunk) > % bzr branch lp:python/2.6 > % bzr branch lp:python/py3k > % bzr branch lp:python/3.1 > > (It's fairly easy to create new mirrors for other Subversion branches, > e.g. Python 2.5; just drop me an email if you want them.) > > If you're going to create a lot of branches you probably want to put them > in a > shared repository. E.g. > > % bzr init-repo pythonbzr > % cd pythonbzr > % bzr branch lp:python/py3k > > Bazaar 2.0 or better is recommended. For me, it took about 5m to check > the > first branch out from Launchpad, and then about 30s or so for each > subsequent > branch. > > Enjoy, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/david.lyon%40pythontest.org > From barry at python.org Wed Jan 20 00:54:39 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 18:54:39 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119185439.299660c7@freewill> On Jan 20, 2010, at 10:16 AM, David Lyon wrote: >Hi Barry, > >That looks very interesting... Hi David, >So does that mean we could update the stdlib for a given >python version using this ? In a sense, yes (if I understand your question correctly). You can use Bazaar to branch any of the 4 Python series and use all the modern DVCS goodness to develop your updates. You can share your branches with others via e.g. Launchpad and even request reviews (called "merge proposals") to get feedback from others. The one thing I am unsure about, mostly because I have not tried it, is whether your Bazaar branch can be used to commit directly back to the Python Subversion master branches. I /think/ the answer is yes, assuming of course that you have permission to do so, and that you have a modern version of Bazaar and the bzr-svn plugin. It might even be possible to commit your Bazaar branch to a local Subversion branch, and then commit the latter to get it pushed up to svn.python.org. At worst, you would use Bazaar's features to get your patch into a state you and your reviewers are happy with, then you would generate a diff for application to your copy of the svn.python.org branch. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 01:51:23 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 11:51:23 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119185439.299660c7@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> Message-ID: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> > On Jan 20, 2010, at 10:16 AM, Barry wrote: >> So does that mean we could update the stdlib for a given >> python version using this ? > > In a sense, yes (if I understand your question correctly). Yeah, it just needs an implementation. > The one thing I am unsure about, mostly because I have not tried it, is > whether your Bazaar branch can be used to commit directly back to the > Python Subversion master branches. I /think/ the answer is yes, > assuming of course that you have permission to do so... Well I'm too Senior and my stuff is too forward looking to qualify for that just yet. I'd be happy to see bzr and mercurial and git all made it together into the stdlib for python 3. That would give a superb updating mechanism for python that would propel python well beyond the dinosaur badlands of CPAN and other languages. I was actually reading from (http://en.wikipedia.org/wiki/Python_%28programming_language%29): "Rather than requiring all desired functionality to be built into the language's core, Python was designed to be highly extensible. .. .. This design of a small core language with a large standard library and an easily extensible interpreter was intended by Van Rossum from the very start because of his frustrations with ABC (which espoused the opposite mindset).[5]" To me, the source code control systems seem to be fully in tune with the original design of python. That is, to be able to easily pull external libraries in. I think what has changed is that the mechanisms now (the SCMs) are way more highly developed than before. Apart from that though, after reading the full wikipedia article I'm left with the distinct impression that things are still pretty much the same (in that python design philosophy is advanced), just that the landscape (of external C libraries) has changed. Now all the libraries are external (on the internet) and all externally managed. So with just a tiny amount of work, imho we could pull it all together to bring python 3 *back* to being that cool tool that it once was (not saying it isn't now). Were you offering me an experimental branch somewhere for python 3 SCM integration ? David From jnoller at gmail.com Wed Jan 20 02:09:15 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 20:09:15 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> On Tue, Jan 19, 2010 at 7:51 PM, David Lyon wrote: >> On Jan 20, 2010, at 10:16 AM, Barry wrote: > >>> So does that mean we could update the stdlib for a given >>> python version using this ? >> >> In a sense, yes (if I understand your question correctly). > > Yeah, it just needs an implementation. > >> The one thing I am unsure about, mostly because I have not tried it, is >> whether your Bazaar branch can be used to commit directly back to the >> Python Subversion master branches. ?I /think/ the answer is yes, >> assuming of course that you have permission to do so... > > Well I'm too Senior and my stuff is too forward looking to qualify > for that just yet. > > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. i sincerely doubt that a source control system will be included in the standard library in the future. Especially 3. A SCM is not a "package management system". Barry was talking about mirrors of the python code. It is true a "package manager" could be developed based on a SCM, however you need to implement this far away from the stdlib and get traction with it within the community long before inclusion would be considered. The decision to move python's source control from SVN to mercurial was controversial enough; including 3 or more scm libraries into core would be an intractable uphill mountain of bike sheds. > So with just a tiny amount of work, imho we could pull > it all together to bring python 3 *back* to being that > cool tool that it once was (not saying it isn't now). Python 3 is still modularized, still has a standard library, etc. If you're really interested in helping with the standard library, get on stdlib-sig, and get ready to write code and PEPs. > Were you offering me an experimental branch somewhere > for python 3 SCM integration ? Barry made bzr mirrors of the python svn tree. Not a python with bzr included. jesse From barry at python.org Wed Jan 20 04:32:30 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 22:32:30 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <20100119223230.4c4a7ed5@freewill> On Jan 19, 2010, at 08:09 PM, Jesse Noller wrote: >The decision to move python's source control from SVN to mercurial was >controversial enough; including 3 or more scm libraries into core >would be an intractable uphill mountain of bike sheds. I'd be surprised if any of the big 3 DVCS developers would actually /want/ their stuff in the stdlib. Being in the stdlib has its advantages and disadvantages. I think for rapidly developing technology, the latter can actually outweigh the former. (Besides, git in the stdlib doesn't make much sense :). >Barry made bzr mirrors of the python svn tree. Not a python with bzr included. Bingo. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 04:43:12 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 14:43:12 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> Message-ID: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> > On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: > Python 3 is still modularized, still has a standard library, etc. If > you're really interested in helping with the standard library, get on > stdlib-sig, and get ready to write code and PEPs. Thank you for your direction to move these items forward to PEPs and Code. > i sincerely doubt that a source control system will be included in the > standard library in the future. Especially 3. Yeah and who twenty years ago thought you would get a 1GB memory card for $3 when all we had was 10Meg hard disks and they were the full 8" platter. > A SCM is not a "package management system". Exactly. It almost makes the need for a "package management system" pretty much obsolete if you can update your code directly from the developers sources. That's what all these SCMs provide. Plus it's addictive. It's hard to go back to 'package' style technology once you have all your code on an SCM based feed. > Barry was talking about mirrors of the python code. It is true a > "package manager" could be developed based on a SCM, however you need > to implement this far away from the stdlib and get traction with it > within the community long before inclusion would be considered. I think I'll have better chances with PEPs. Being honest, if wonderful libraries like Sphinx and Mercurial and Git and BZR can't make it into the stdlib, then there is no hope for even newer code to get in there. Plus, promoting all sorts of new and fangled tools however good they may or may not be just confuses users and ends up being a waste of time imho. It isn't good management of volounteers time and effort. If you could imagine disaster relief coordinated this way, it would just be a disaster in itself. That's why it has taken some 5 years to get PEP-345 done. > The decision to move python's source control from SVN to mercurial was > controversial enough; including 3 or more scm libraries into core > would be an intractable uphill mountain of bike sheds. Not at all. It would be a very fair thing to do. Not to mention being great for users. > Barry made bzr mirrors of the python svn tree. Not a python with bzr > included. I can't resist asking for that again.. I heard it only in Monty speek. Did you just say ?: "Barry made a bizarre mirror of the python suvern tree. Not a python with a buzzer included." Anyway.. Maybe I do get what your talking about. Even if you do talk with a strange accent. :-) David From jnoller at gmail.com Wed Jan 20 05:10:07 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Jan 2010 23:10:07 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8491001192010v66b728dbxa7ad1ed1fc1df15f@mail.gmail.com> On Tue, Jan 19, 2010 at 10:43 PM, David Lyon wrote: [snip] > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. Did you ever stop to think that some package authors do not want their code in the standard library? That throwing random shiny things in there just makes it a junk drawer? Besides: Show us the PEP to include Sphinx, and it's dependencies in the standard lib, with Georg's signature at the bottom. The authors of modules have to want things to be in there, they have to be best of breed, tested or common-enough patterns to warrant a slot. We have things in the standard lib which still need more TLC and maintenance, or they need to be shunted into space. Adding random things, which may or may not help packaging, and installing things from random places in someone source repository won't make things better. > Plus, promoting all sorts of new and fangled tools however > good they may or may not be just confuses users and ends > up being a waste of time imho. It isn't good management > of volounteers time and effort. > > If you could imagine disaster relief coordinated this > way, it would just be a disaster in itself. > > That's why it has taken some 5 years to get PEP-345 done. I'm going to assume that you're trolling now, or intentionally misrepresenting facts. Maybe a little of both. A PEP, and an implementation and the ability to rationally debate, discuss and defend your proposal is what is needed to enact changes on policies, python-core or the standard library. >> The decision to move python's source control from SVN to mercurial was >> controversial enough; including 3 or more scm libraries into core >> would be an intractable uphill mountain of bike sheds. > > Not at all. > > It would be a very fair thing to do. Not to mention being > great for users. There should be one-- and preferably only one --obvious way to do it. > Anyway.. Maybe I do get what your talking about. Even if you do > talk with a strange accent. :-) My sense of humor has been disabled by repeated stunning at your hands. I admire your enthusiasm, even if I do think some of it is misplaced, or at guided into the proper channels at very least. Please, you seem to have the time and willingness to help, please go about this the right way. Discuss things on the proper lists, make concrete proposals. If you have have standard lib changes, discuss them on stdlib-sig, if you have ideas about python-the-language, or the interpreter, etc - please discuss it on python-ideas. jesse From ben+python at benfinney.id.au Wed Jan 20 05:16:25 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 15:16:25 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <87wrzdfm1y.fsf@benfinney.id.au> "David Lyon" writes: > Being honest, if wonderful libraries like Sphinx and Mercurial and Git > and BZR can't make it into the stdlib, then there is no hope for even > newer code to get in there. Those are applications, not libraries. Applications don't belong in the standard library. -- \ ?If you pick up a starving dog and make him prosperous, he will | `\ not bite you. This is the principal difference between a dog | _o__) and a man.? ?Mark Twain, _Pudd'n'head Wilson_ | Ben Finney From brian.curtin at gmail.com Wed Jan 20 05:19:47 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 19 Jan 2010 22:19:47 -0600 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: On Tue, Jan 19, 2010 at 21:43, David Lyon wrote: > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > I'm not entirely sure I see why the inclusion of a SCM into the stdlib is necessary. Just because pieces of software are mature and proven in their fields doesn't mean we should add them, or that them *not* being in the stdlib should be a basis for other projects making it in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Jan 20 05:26:44 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:26:44 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119232644.7fa25691@freewill> On Jan 20, 2010, at 02:43 PM, David Lyon wrote: >> On Tue, Jan 19, 2010 at 7:51 PM, Jesse Noller wrote: >> A SCM is not a "package management system". > >Exactly. It almost makes the need for a "package management system" >pretty much obsolete if you can update your code directly from >the developers sources. > >That's what all these SCMs provide. Plus it's addictive. It's >hard to go back to 'package' style technology once you have >all your code on an SCM based feed. Well... I'm not so sure. A package management system like apt does a /ton/ of additional bookkeeping and work to ensure a robust, highly consistent, functioning system. And while both Python and most Linux distributions have their own notion of "package management", they don't always play nicely together. Tarek and the distutils-sig's work is trying to make the world a better place by bridging this gap better, and there is code out there that makes it easier to say import a Python package from the Cheeseshop and .deb-ify it for use on Debian and Ubuntu. There's also work being done in Launchpad that will allow you to "build-from-branch" so that in a sense you could let a build farm take your Bazaar branches and automatically build the packages from them. I've strayed off-topic I suppose, but I see SCMs and package managers as complementary technologies that help with important parts of the process of delivering software to end-users, but I don't quite see how one can make the other obsolete. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 05:29:34 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 15:29:34 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> > On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: > > I'd be surprised if any of the big 3 DVCS developers would actually /want/ > their stuff in the stdlib. If they ask, they'll get told they're motorbike-shedding. "It's better if their users ask". So here I am as a user doing things the 'right' way. > Being in the stdlib has its advantages and > disadvantages. I think for rapidly developing technology, the > latter can actually outweigh the former. If it's about being able to do updates, then I think this resolves an old and circular argument. As the SCM implementation would, one would expect, to be able to update itself. Side benefits are that it can update everything else along with it at the same time. User Apps, Packages, whatever. It's even better having SCM in an Industrial/Scientific environment. Here's an example: - a machine breaks.. (I mean the software for/in it) - you fix the code, maybe on the spot - you commit and push back to the repository - your code gets checked in and run through the testbot and then you get blamed and have to do the whole thing again properly with a test case. Oh well.. Well anyway, whatever you guys might say, that's a whole lot more efficient than running back to the development machine and going through some obscure build and test and publish process to do a fix on a production machine. Point : The fact that SCMs are two way is great in a production environment. No packaging solution can come close. So why not have python SCMs included as batteries in python.. All these arguments I can take off to the stdlib list when I get the chance.. David From barry at python.org Wed Jan 20 05:50:36 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 19 Jan 2010 23:50:36 -0500 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> Message-ID: <20100119235036.57f6e867@freewill> Okay, last follow up on this and then I'm going to bed. :) On Jan 20, 2010, at 03:29 PM, David Lyon wrote: >> On Jan 19, 2010, at 08:09 PM, Barry Warsaw wrote: >> >> I'd be surprised if any of the big 3 DVCS developers would actually /want/ >> their stuff in the stdlib. > >If they ask, they'll get told they're motorbike-shedding. "It's better >if their users ask". So here I am as a user doing things the 'right' >way. Actually, you're not. It's not up to the Python community to initiate this. If you really want this, you should engage with the relevant DVCS communities and push them to request it. >Side benefits are that it can update everything else along >with it at the same time. User Apps, Packages, whatever. I get that. Heck, I still run one Gentoo server which I think is as close to the edge you're describing as I'm comfortable with. It's all great until the wheels come off and then it can take *days* to get a functioning system again. The big difference is that I rely on my DVCS to keep one small thing, or a few variants of the same thing, all sane. But I rely on my distribution vendor to keep a thousand complex, interdependent, interacting, sometimes conflicting things sane and working. >Point : The fact that SCMs are two way is great in > a production environment. No packaging solution > can come close. Try talking with some hard-core operations guys, the folks with the keys to the data centers, who work tireless, insanely hours keeping incredibly complex systems running with very little downtime. I think you'd get a different perspective to put it mildly. :) to-sleep-perchance-to-dream-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From david.lyon at pythontest.org Wed Jan 20 06:10:15 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 16:10:15 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <87wrzdfm1y.fsf@benfinney.id.au> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> Message-ID: <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> > "David Lyon" writes: > >> Being honest, if wonderful libraries like Sphinx and Mercurial and Git >> and BZR can't make it into the stdlib, then there is no hope for even >> newer code to get in there. > > Those are applications, not libraries. Applications don't belong in the > standard library. Haha funny.. Well using that logic, distutils is an application.. Are you saying that distutils should be removed? That is most certainly an application. Lets not get too pedantic here. Mercurial and bzr have a built in API that can be called in a library like way. It's true they also have a command line interface in the same way that distutils does. I'm not saying anything negative about distutils. Given that Tarek has an upcoming Pycon presentation where the program talks about a distutils revamp. I'm hoping that he can find some young 20 yr olds and put a cool web interface on that thing. Given that there are empty sprints at pycon. It couldn't hurt to throw that challenge out. Anyway, we'll see.. David From ben+python at benfinney.id.au Wed Jan 20 06:32:10 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:32:10 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> <1669.218.214.45.58.1263961774.squirrel@syd-srv02.ezyreg.com> <20100119235036.57f6e867@freewill> Message-ID: <87iqaxfijp.fsf@benfinney.id.au> Barry Warsaw writes: > On Jan 20, 2010, at 03:29 PM, David Lyon wrote: > >So here I am as a user doing things the 'right' way. > > Actually, you're not. It's not up to the Python community to initiate > this. If you really want this, you should engage with the relevant > DVCS communities and push them to request it. Where ?push? must be strictly limited by a continual awareness that the whole idea could just be bad. If you find yourself in a tiny minority pushing for a change, it *could* be that you have a great idea and the vast majority don't realise it yet. But you must be realistic about the likelihood that the change is a very *bad* idea, and frequently evaluate it for signs of that. -- \ ?I used to think that the brain was the most wonderful organ in | `\ my body. Then I realized who was telling me this.? ?Emo Philips | _o__) | Ben Finney From ben+python at benfinney.id.au Wed Jan 20 06:34:07 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Jan 2010 16:34:07 +1100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> <87wrzdfm1y.fsf@benfinney.id.au> <1695.218.214.45.58.1263964215.squirrel@syd-srv02.ezyreg.com> Message-ID: <87eillfigg.fsf@benfinney.id.au> "David Lyon" writes: > Well using that logic, distutils is an application.. Distutils is an application, the function of which is essential to allowing sane development of Python packages. It's a special case. We need to strictly limit the number of special cases, not gleefully add to them. -- \ ?I'd take the awe of understanding over the awe of ignorance | `\ any day.? ?Douglas Adams | _o__) | Ben Finney From matthieu.brucher at gmail.com Wed Jan 20 07:49:36 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 07:49:36 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: > I'd be happy to see bzr and mercurial and git all made it together > into the stdlib for python 3. That would give a superb updating > mechanism for python that would propel python well beyond > the dinosaur badlands of CPAN and other languages. I think there are several points that make them not includable in Python: - git is not written in Python - bzr and mercurial have a life cycle much shorter than Python's, it's the same issue than with other libraries where another community develops them. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From david.lyon at pythontest.org Wed Jan 20 08:20:02 2010 From: david.lyon at pythontest.org (David Lyon) Date: Wed, 20 Jan 2010 18:20:02 +1100 (EST) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> Message-ID: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Matthieu, >> I'd be happy to see bzr and mercurial and git all made it together >> into the stdlib for python 3. That would give a superb updating >> mechanism for python that would propel python well beyond >> the dinosaur badlands of CPAN and other languages. > > I think there are several points that make them not includable in Python: > - git is not written in Python > - bzr and mercurial have a life cycle much shorter than Python's, it's > the same issue than with other libraries where another community > develops them. That's only two points. :-) On 1; If that's true, I won't mention git again. On 2; Who knows what their life cycle is. CVS is pretty much dead, and svn looks like it is on the way out. I can't think of how anything could be better than mercurial or bzr but I know I will be proved wrong. At the end of the day, we are making a decision about whether the language is 'set-in-stone' or whether it is still evolving. To me, Python 1.x had it's own distinct "era", as has Python 2.x Hoping that the Python 3 era can be a little more flexible and perphaps "cleaner" than the 2.x era is all that I am thinking here. Have a nice day David From matthieu.brucher at gmail.com Wed Jan 20 09:05:19 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 20 Jan 2010 09:05:19 +0100 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: > That's only two points. :-) In French, we say that several starts with 2 ;) > On 1; If that's true, I won't mention git again. I tis, you can check on the git repository (it's a mix of C, perl, shell scripts, Python, ...) > On 2; Who knows what their life cycle is. You can check on their websites, their cycles are far shorter than Python minor releases (several months vs several years). Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From stephen at xemacs.org Wed Jan 20 11:22:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 20 Jan 2010 19:22:55 +0900 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <20100119223230.4c4a7ed5@freewill> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <20100119223230.4c4a7ed5@freewill> Message-ID: <87aaw9gjnk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > (Besides, git in the stdlib doesn't make much sense :). "Dulwich." From solipsis at pitrou.net Wed Jan 20 12:28:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Jan 2010 11:28:16 +0000 (UTC) Subject: [Python-Dev] Bazaar branches available (again) on Launchpad References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <4222a8491001191709m38f1cb96if5e546ed7dc008ab@mail.gmail.com> <1622.218.214.45.58.1263958992.squirrel@syd-srv02.ezyreg.com> Message-ID: David Lyon pythontest.org> writes: > > I think I'll have better chances with PEPs. > > Being honest, if wonderful libraries like Sphinx and Mercurial > and Git and BZR can't make it into the stdlib, then there is > no hope for even newer code to get in there. > [snip] This is python-ideas material, can you take it there? Thank you. Antoine. From ncoghlan at gmail.com Wed Jan 20 13:16:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Jan 2010 22:16:34 +1000 Subject: [Python-Dev] Bazaar branches available (again) on Launchpad In-Reply-To: <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> References: <20100119081639.5d431ed9@freewill> <1377.218.214.45.58.1263943004.squirrel@syd-srv02.ezyreg.com> <20100119185439.299660c7@freewill> <1488.218.214.45.58.1263948683.squirrel@syd-srv02.ezyreg.com> <47475.115.128.47.203.1263972002.squirrel@syd-srv02.ezyreg.com> Message-ID: <4B56F422.5010607@gmail.com> David Lyon wrote: > On 2; Who knows what their life cycle is. CVS is pretty much > dead, and svn looks like it is on the way out. > I can't think of how anything could be better than > mercurial or bzr but I know I will be proved wrong. I believe you misunderstood what Matthieu meant by life cycle there: think "release cycle". If a project pushes out new releases significantly more often than every 18-24 months (as is currently true for all of the major SCM tools), then that fact alone makes it a very bad fit for the Python standard library. And centralised source control will be going strong for years. The DVCS approach may be great for the open source world, but the gains are far more limited in a closed source shop (especially a group writing internal corporate applications which doesn't need to keep many, if any, maintenance branches going). If we weren't dealing with 4 active branches, the DVCS discussion would have got a lot less traction with the core developers - aside from better handling of multiple lines of development, most of the benefits of the switch to a DVCS accrue to people without commit access to the SVN repository. Anyway, we've wandered far afield from legit python-dev topics now. Any further ideas about super_mega_easy_install functionality that can pull code from source control systems and build it rather than requiring prebuild source tarballs should be directed to python-ideas (they probably need to bake more even before they make an appearance on distutils-sig). Cheers, Nick. P.S. As Jesse said... your enthusiasm is great, but please don't assume that some inherent conservatism on the part of other developers is automatically evil or the result of a failure to see your point. A lot of people around the world rely on our stuff every day. We owe it to them to be measured in our actions and to put serious thought into any major changes or additions we make to the language and the standard library. For the current stage of its development, Python 3 is in a good place from our point of view - its major carrot has really always been the better Unicode support it offers, and the ever-increasing globalisation of the web will create more and more pressure pushing developers in that direction as the years go by. Sure, Python 3 cleans up assorted other things as well, but the change to the text processing model is the big one that is fundamentally incompatible with the architecture of the 2.x series. Compared to that change, everything else is just tinkering. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ---------------------------------------------------------------