From techtonik at gmail.com Thu Apr 1 11:05:44 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 1 Apr 2010 12:05:44 +0300 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha Message-ID: I can not compile Python itself, so I use Alpha version to run tests in trunk. Recent update broke successfully running tests. Any hints why this happened and how to fix them back? > C:\~env\Python27\python.exe test\test_doctest.py doctest (doctest) ... 66 tests with zero failures Traceback (most recent call last): File "test\test_doctest.py", line 2492, in test_main() File "test\test_doctest.py", line 2474, in test_main with test_support.check_warnings(*deprecations): File "C:\~env\Python27\lib\contextlib.py", line 84, in helper return GeneratorContextManager(func(*args, **kwds)) TypeError: check_warnings() takes no arguments (1 given) -- anatoly t. From screwdriver at lxnt.info Thu Apr 1 11:49:43 2010 From: screwdriver at lxnt.info (Alexander Sabourenkov) Date: Thu, 01 Apr 2010 13:49:43 +0400 Subject: [Python-Dev] Replacing threads with swapcontext() In-Reply-To: <4BB3C46E.8000507@gmail.com> References: <4BB368EF.8010206@lxnt.info> <4BB3C46E.8000507@gmail.com> Message-ID: <4BB46C37.8010401@lxnt.info> Nick Coghlan wrote: > Off the top of my head, recursion limit checks will definitely go > haywire, and I would expect threading.local() and exception handling to > explode at some point (although the latter depends on what you mean by > "switching C stacks" - if you really are just fiddling the stack pointer > without affecting the thread local heap storage mechanisms, the latter > two should be fine. Still doesn't sound like a good idea though). In my implementation, each context looks like a thread to Python, with its own PyThreadState, thread-local storage and locks are taken care of, so I'm reasonably sure there's nothing scary lurking unseen. I'm trying to confirm my understanding as to how unreliable the greenlet is. The greenlet module basically just switches C stacks and does not touch anything else. There evidently are problems with this when there's an exception raised, as it's stored in PyThreadState::curexc_*/exc_*. PyThreadState's profiling/tracing -related members, the tick_counter, and per-thread state dict don't appear as a problem at the first glance. recursion_depth will be messed up unpredictably, as you kindly pointed out. This leaves PyThreadState::frame a PyFrameObject to be examined. Do they form a python-side stack of a thread in question? -- ./lxnt From gregg.lind at gmail.com Thu Apr 1 05:38:39 2010 From: gregg.lind at gmail.com (Gregg Lind) Date: Wed, 31 Mar 2010 22:38:39 -0500 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: Message-ID: Thank you for the advice everyone. This seed has finally born (rotten) fruit at: http://writeonly.wordpress.com/2010/04/01/whython-python-for-people-who-hate-whitespace/ http://bitbucket.org/gregglind/python-whython3k/ On Fri, Mar 12, 2010 at 4:13 AM, Georg Brandl wrote: > Am 09.03.2010 14:42, schrieb Jeremy Hylton: > > On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind > wrote: > >> Python-devs, > >> > >> I'm writing to you for some help in understanding the Python grammar. > As an > >> excuse to deep dive into Python's tokenizer / grammar, I decided (as a > >> hideous, hideous joke) to want to allow braces where colons are allowed > (as > >> flow control). > >> > >> Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer > >> > >> As a first example: > >> > >> funcdef: ('def' NAME parameters ['->' test] ':' suite | > >> 'def' NAME parameters ['->' test] '{' suite '}' ) > >> > >> I reran Parser/pgen and the dfa changes, but python (3.1) when > recompiled, > >> throws errors on things like: > >> > >> def a() { None } > >> > >> Strangely enough: > >> > >> lambdef: ( 'lambda' [varargslist] ':' test | > >> 'lambda' [varargslist] '{' test '}' ) > >> > >> works fine! I this simplely some difference between "test" and "suite". > >> > >> I have tried tackling this with gdb, looking at err_input clearly isn't > >> enough. > >> > >> (gdb) break err_input > >> (gdb) break PyParser_ASTFromString > >> import sys > >> b = compile("def a() {pass}","sys.stdout","single") > >> # yet a simple grammar fix is enough for this! > >> c = compile("lambda x {None}","sys.stdout","single") > >> > >> I'm in over my head! > > > > You don't say what errors occur when you try to compile strings in > > your new language. You may have changed the Grammar, which allows you > > to tokenize the input. That isn't enough to get the input to compile. > > You also need to change the compiler to understand the new tokens. > > In particular, many AST creation functions check for specific counts of > children on many nodes. I haven't checked, but in the case of the > "funcdef" rule, it may check for either 7 or 5 children to determine > whether the optional return annotation ['->' test] is present. > > Georg > > > -- > Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. > Four shall be the number of spaces thou shalt indent, and the number of thy > indenting shall be four. Eight shalt thou not indent, nor either indent > thou > two, excepting that thou then proceed to four. Tabs are right out. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/gregg.lind%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu Apr 1 13:18:08 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 01 Apr 2010 12:18:08 +0100 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha In-Reply-To: References: Message-ID: <4BB480F0.7020909@voidspace.org.uk> On 01/04/2010 10:05, anatoly techtonik wrote: > I can not compile Python itself, Building Python on Windows can be done with free tools, so it should be possible for you to build Python. See the instructions here: http://python.org/dev/faq/#id8 > so I use Alpha version to run tests > in trunk. Recent update broke successfully running tests. Any hints > why this happened and how to fix them back? > > >> C:\~env\Python27\python.exe test\test_doctest.py >> > doctest (doctest) ... 66 tests with zero failures > Traceback (most recent call last): > File "test\test_doctest.py", line 2492, in > test_main() > File "test\test_doctest.py", line 2474, in test_main > with test_support.check_warnings(*deprecations): > File "C:\~env\Python27\lib\contextlib.py", line 84, in helper > return GeneratorContextManager(func(*args, **kwds)) > TypeError: check_warnings() takes no arguments (1 given) > > When I run this test with a freshly built Python I get the following: :\compile\python-trunk\PCbuild > python_d.exe ..\Lib\test\test_doctest.py doctest (doctest) ... 66 tests with zero failures doctest (test.test_doctest) ... 428 tests with zero failures [42795 refs] All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From techtonik at gmail.com Thu Apr 1 14:15:19 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 1 Apr 2010 15:15:19 +0300 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha In-Reply-To: <4BB480F0.7020909@voidspace.org.uk> References: <4BB480F0.7020909@voidspace.org.uk> Message-ID: Thanks. I've copied test/test_support.py form Lib into 2.7 alpha directory and it seems to work. Although it doesn't seem good to me to mix test support library with tests themselves. -- anatoly t. On Thu, Apr 1, 2010 at 2:18 PM, Michael Foord wrote: > On 01/04/2010 10:05, anatoly techtonik wrote: >> >> I can not compile Python itself, > > Building Python on Windows can be done with free tools, so it should be > possible for you to build Python. > > See the instructions here: > > http://python.org/dev/faq/#id8 > >> so I use Alpha version to run tests >> in trunk. Recent update broke successfully running tests. Any hints >> why this happened and how to fix them back? >> >> >>> >>> C:\~env\Python27\python.exe test\test_doctest.py >>> >> >> doctest (doctest) ... 66 tests with zero failures >> Traceback (most recent call last): >> ? File "test\test_doctest.py", line 2492, in >> ? ? test_main() >> ? File "test\test_doctest.py", line 2474, in test_main >> ? ? with test_support.check_warnings(*deprecations): >> ? File "C:\~env\Python27\lib\contextlib.py", line 84, in helper >> ? ? return GeneratorContextManager(func(*args, **kwds)) >> TypeError: check_warnings() takes no arguments (1 given) >> >> > > When I run this test with a freshly built Python I get the following: > > :\compile\python-trunk\PCbuild >> python_d.exe ..\Lib\test\test_doctest.py > doctest (doctest) ... 66 tests with zero failures > doctest (test.test_doctest) ... 428 tests with zero failures > [42795 refs] > > All the best, > > > Michael Foord > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > From fuzzyman at voidspace.org.uk Thu Apr 1 14:59:27 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 01 Apr 2010 13:59:27 +0100 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha In-Reply-To: References: <4BB480F0.7020909@voidspace.org.uk> Message-ID: <4BB498AF.4000004@voidspace.org.uk> On 01/04/2010 13:15, anatoly techtonik wrote: > Thanks. I've copied test/test_support.py form Lib into 2.7 alpha > directory and it seems to work. > Although it doesn't seem good to me to mix test support library with > tests themselves. > What do you mean by "it doesn't seem good to me to mix test support library with tests themselves"? Do you mean to have it in the same directory - where would you put it? It isn't *meant* to be a public library, it exists only to support the test framework. In Python 3 it has been renamed support.py, but lives in the same location. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From techtonik at gmail.com Thu Apr 1 16:33:42 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 1 Apr 2010 17:33:42 +0300 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha In-Reply-To: <4BB498AF.4000004@voidspace.org.uk> References: <4BB480F0.7020909@voidspace.org.uk> <4BB498AF.4000004@voidspace.org.uk> Message-ID: On Thu, Apr 1, 2010 at 3:59 PM, Michael Foord wrote: >> >> Thanks. I've copied test/test_support.py form Lib into 2.7 alpha >> directory and it seems to work. >> Although it doesn't seem good to me to mix test support library with >> tests themselves. >> > > What do you mean by "it doesn't seem good to me to mix test support library > with tests themselves"? Do you mean to have it in the same directory - where > would you put it? It isn't *meant* to be a public library, it exists only to > support the test framework. In Python 3 it has been renamed support.py, but > lives in the same location. I mean that usually testing tools/libraries are separated from tests itself, as well as data. test_support.py is not only located in the same directory - it is even named in the same way. The test directory looks like a mess with all these aux data files. But it is hard to estimate if it would be worthy to separate testing framework from tests and test data. It may happen that writing and debugging tests become harder, because Python is not locked into some specific usage domain. -- anatoly t. From ncoghlan at gmail.com Thu Apr 1 17:53:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Apr 2010 01:53:07 +1000 Subject: [Python-Dev] trunk doctests fail to execute with 2.7 alpha In-Reply-To: References: <4BB480F0.7020909@voidspace.org.uk> <4BB498AF.4000004@voidspace.org.uk> Message-ID: <4BB4C163.9090406@gmail.com> anatoly techtonik wrote: > I mean that usually testing tools/libraries are separated from tests > itself, as well as data. test_support.py is not only located in the > same directory - it is even named in the same way. The test directory > looks like a mess with all these aux data files. But it is hard to > estimate if it would be worthy to separate testing framework from > tests and test data. It may happen that writing and debugging tests > become harder, because Python is not locked into some specific usage > domain. Tests start with test_*, the support files don't. The only odd one out was test_support, and that has been fixed for 3.x. The generalised test frameworks (unittest, doctest) do live in the standard library. It's only the stuff specific to *our* unit tests that lives in the test directory (and certainly, things from that directory will sometimes get generalised and moved to the standard library - that's how warning.catch_warnings was created). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Apr 1 17:57:43 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Apr 2010 01:57:43 +1000 Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts In-Reply-To: References: Message-ID: <4BB4C277.2050001@gmail.com> Gregg Lind wrote: > Thank you for the advice everyone. This seed has finally born (rotten) > fruit at: > > http://writeonly.wordpress.com/2010/04/01/whython-python-for-people-who-hate-whitespace/ > http://bitbucket.org/gregglind/python-whython3k/ Nicely done :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From techtonik at gmail.com Thu Apr 1 18:40:19 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 1 Apr 2010 19:40:19 +0300 Subject: [Python-Dev] skip all TestCase methods if resource is not available Message-ID: Currently it is possible to mark individual test methods with: test_support.requires('network') However, sometimes it is necessary to skip the whole TestCase if 'network' resource is not available counting the number of skipped tests at the same time. Are there any standard means to do this? -- anatoly t. From florent.xicluna at gmail.com Thu Apr 1 19:02:38 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Thu, 1 Apr 2010 19:02:38 +0200 Subject: [Python-Dev] skip all TestCase methods if resource is not available In-Reply-To: References: Message-ID: 2010/4/1 anatoly techtonik: > Currently it is possible to mark individual test methods with: > test_support.requires('network') > > However, sometimes it is necessary to skip the whole TestCase if > 'network' resource is not available counting the number of skipped > tests at the same time. Are there any standard means to do this? Put it in unittest.TestCase.setUp() method. It should be enough. -- Florent From techtonik at gmail.com Thu Apr 1 20:16:02 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 1 Apr 2010 21:16:02 +0300 Subject: [Python-Dev] skip all TestCase methods if resource is not available In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 8:02 PM, Florent Xicluna wrote: > 2010/4/1 anatoly techtonik: >> Currently it is possible to mark individual test methods with: >> ? ? ? ? ? ?test_support.requires('network') >> >> However, sometimes it is necessary to skip the whole TestCase if >> 'network' resource is not available counting the number of skipped >> tests at the same time. Are there any standard means to do this? > > Put it in unittest.TestCase.setUp() method. It should be enough. It fails with error instead if skip, as it should according to http://docs.python.org/library/unittest.html#unittest.TestCase.setUp "...any exception raised by this method will be considered an error rather than a test failure..." -- anatoly t. From florent.xicluna at gmail.com Thu Apr 1 21:03:09 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Thu, 1 Apr 2010 21:03:09 +0200 Subject: [Python-Dev] skip all TestCase methods if resource is not available In-Reply-To: References: Message-ID: 2010/4/1 anatoly techtonik: > On Thu, Apr 1, 2010 at 8:02 PM, Florent Xicluna wrote: (...) >> >> Put it in unittest.TestCase.setUp() method. It should be enough. > > It fails with error instead if skip, as it should according to > http://docs.python.org/library/unittest.html#unittest.TestCase.setUp > > "...any exception raised by this method will be considered an error > rather than a test failure..." > -- > anatoly t. > There's a special case for the "SkipTest" exception in 2.7 (even if it is not documented). try: self.setUp() except SkipTest as e: self._addSkip(result, str(e)) except Exception: result.addError(self, sys.exc_info()) But for 2.6, you're right: try: self.setUp() except: result.addError(self, self._exc_info()) -- Florent From barry at python.org Thu Apr 1 22:12:42 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Apr 2010 16:12:42 -0400 Subject: [Python-Dev] PEP 3147 working implementation Message-ID: <20100401161242.6405053d@heresy> I now have a working implementation of PEP 3147 which passes all the existing, and new, tests. I'm sure there's still work to do, but I think the branch is in good enough shape to start getting some feedback from python-dev. You can grab the changes in several ways. If you have Bazaar, you can check out the branch by doing: % bzr branch lp:~barry/python/pep3147 You can also view a live diff online: https://code.launchpad.net/~barry/python/pep3147/+merge/22648 or just download the diff, which should apply cleanly against the py3k Subversion branch (or pretty close): https://code.launchpad.net/~barry/python/pep3147/+merge/22648/+preview-diff/+files/preview.diff Modulo some lag time, the diffs should track changes I make to the branch. You can provide feedback to me directly, as a follow up to this message, or in Launchpad. I've also updated the PEP to point to these locations for the reference implementation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Thu Apr 1 23:27:36 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Apr 2010 21:27:36 +0000 (UTC) Subject: [Python-Dev] PEP 3147 working implementation References: <20100401161242.6405053d@heresy> Message-ID: Barry Warsaw python.org> writes: > > You can also view a live diff online: > > https://code.launchpad.net/~barry/python/pep3147/+merge/22648 If you want a review, perhaps you should post it to Rietveld. Regards Antoine. From ziade.tarek at gmail.com Thu Apr 1 23:51:23 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 1 Apr 2010 23:51:23 +0200 Subject: [Python-Dev] Proposing PEP 376 Message-ID: Hello, On behalf of the distutils-SIG, I'd like to propose PEP 376 for inclusion. This PEP is about defining a standard for installed Python projects. http://www.python.org/dev/peps/pep-0376/ It was created : - to allow interoperability among all package managers. - to provide a set of APIs in the stdlib to know what's installed (in pkgutil) - to provide a basic *uninstaller* feature in the distutils2 project. It is very similar to the retired PEP 262, but closer to the standard the Setuptools project have created, which is quite used in the community. Calendar-wise, I'd love to see it accepted for 2.7 but it might be too late to finish discussing it (2.7b1 is due in a few days). So I guess the next target will be Python 3.2. One topic has been ignored on purpose in this PEP : how to describe in a detailed way the resources file a project contains, and how they should be installed on a FHS-compliant system (like, a configuration file should go in /etc and such things) We are currently focusing our discussion efforts on the resource files (see the summary work so far here : http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst) and we will produce another PEP for that, that will not interfer with PEP 376. Regards Tarek -- Tarek Ziad? | http://ziade.org From barry at python.org Thu Apr 1 23:53:38 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Apr 2010 17:53:38 -0400 Subject: [Python-Dev] PEP 3147 working implementation In-Reply-To: References: <20100401161242.6405053d@heresy> Message-ID: <20100401175338.1c3fba67@heresy> On Apr 01, 2010, at 09:27 PM, Antoine Pitrou wrote: >If you want a review, perhaps you should post it to Rietveld. Good idea. http://codereview.appspot.com/842043/show I've never used Rietveld before so let me know if I need to do anything to invite you or otherwise make the review possible. I will update the PEP. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From andrew.svetlov at gmail.com Fri Apr 2 01:20:33 2010 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 2 Apr 2010 02:20:33 +0300 Subject: [Python-Dev] copying of itertools iterators Message-ID: using of copy.copy for simple iterators is forbidden >>> import copy >>> copy.copy(iter([1, 2, 3])) Traceback (most recent call last): File "", line 1, in File "/home/andrew/projects/py3k/Lib/copy.py", line 96, in copy return _reconstruct(x, rv, 0) File "/home/andrew/projects/py3k/Lib/copy.py", line 284, in _reconstruct y = callable(*args) File "/home/andrew/projects/py3k/Lib/copyreg.py", line 88, in __newobj__ return cls.__new__(cls, *args) TypeError: object.__new__(list_iterator) is not safe, use list_iterator.__new__() That behavior is safe and clean. But it's possible to copy iterator objects returned by itertools functions: >>> i = itertools.chain([1, 2], [3, 4, 5]) >>> i.__next__() 1 >>> j = copy.copy(i) >>> j.__next__() Traceback (most recent call last): File "", line 1, in StopIteration >>> i.__next__() 2 Looks like itertools object should be protected from usage like that. Folks, what are you think about? From pje at telecommunity.com Fri Apr 2 01:36:53 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 01 Apr 2010 19:36:53 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: Message-ID: <20100401233652.764823A40AF@sparrow.telecommunity.com> At 11:51 PM 4/1/2010 +0200, Tarek Ziad? wrote: >Hello, > >On behalf of the distutils-SIG, I'd like to propose PEP 376 for >inclusion. This PEP is about defining a standard for installed Python >projects. > >http://www.python.org/dev/peps/pep-0376/ Very nice; I have only a few questions/suggestions, and some minor corrections. First, I notice the RECORD paths are listed as being relative to sys.prefix; I was under the impression that these paths were supposed to be relative to the base installation location of the library (i.e. site-packages in the default case). That is, that while paths not under sys.prefix were to be absolute, the paths under sys.prefix were relative to the install location. (Since one of the original motivating use cases for relative paths in RECORD was to support relocatable installations, and sharing installations across platforms, when the code is platform independent code.) Also, the spec doesn't address installation to paths that aren't under sys.prefix, or what happens if you use "setup.py install --prefix". I'm wondering if perhaps this should be restated as something like: * Paths under the base installation location are relative to the base * Paths not under the base installation location, but under the installation prefix, are also stored relative to the base, IF the base location is a subpath of the installation prefix * All other paths are absolute. Where "base location" is the effective --install-lib directory, and prefix is the effective --prefix. (Which default of course to site-package and sys.prefix respectively, but the spec shouldn't be in terms of those defaults.) Second, the RECORD spec is not clear on what happens to the hash field on .pyc/.pyo files. Is it supposed to be an empty string, a hash of an empty file, a hash value that's ignored, or...? Third, what is a "local absolute path"? This term appears several places but is not defined. It sounds like maybe you mean a path using the local platform separator, as opposed to a '/'-separated relative path. If that's the case, this should probably be spelled out somewhere early on and then referenced in context. Fourth, the PEP says the implementation will support zipfiles, but the draft implementation doesn't appear to support zipimport subdirectories. (Zipimport paths can be /path/to/some.zip/subdir, and this base is then used for importing, just as if the zipfile were a normal parent directory.) If this is an intentional omission, it should probably be mentioned in the PEP. Okay, on to the minor corrections. The following uses of "package" should be replaced by "project" or "distribution": "If a package that was already installed on the system as a dependency" "in other words, whether the package was installed by user request" "$ python -m distutils.uninstall packagename" Anyway, that's my first pass through; I think the RECORD section may need further fleshing out (e.g., specifying a particular csv-module dialect) in order to ensure that the spec is sufficiently rigorous for multiple tools to consume/produce compatible files. From raymond.hettinger at gmail.com Fri Apr 2 01:50:08 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 1 Apr 2010 16:50:08 -0700 Subject: [Python-Dev] copying of itertools iterators In-Reply-To: References: Message-ID: <182C94C2-E03F-4EE7-A4EB-9031FEE1FA2D@gmail.com> On Apr 1, 2010, at 4:20 PM, Andrew Svetlov wrote: > using of copy.copy for simple iterators is forbidden > >>>> import copy >>>> copy.copy(iter([1, 2, 3])) > Traceback (most recent call last): > File "", line 1, in > File "/home/andrew/projects/py3k/Lib/copy.py", line 96, in copy > return _reconstruct(x, rv, 0) > File "/home/andrew/projects/py3k/Lib/copy.py", line 284, in _reconstruct > y = callable(*args) > File "/home/andrew/projects/py3k/Lib/copyreg.py", line 88, in __newobj__ > return cls.__new__(cls, *args) > TypeError: object.__new__(list_iterator) is not safe, use > list_iterator.__new__() > > That behavior is safe and clean. > But it's possible to copy iterator objects returned by itertools functions: > >>>> i = itertools.chain([1, 2], [3, 4, 5]) >>>> i.__next__() > 1 >>>> j = copy.copy(i) >>>> j.__next__() > Traceback (most recent call last): > File "", line 1, in > StopIteration >>>> i.__next__() > 2 > > Looks like itertools object should be protected from usage like that. > Folks, what are you think about? > I find it hard to get excited about this. It doesn't seem to have been a problem in the real world (no complaints, bug reports, or feature requests). The tee() itertool is the official way to split-out an iterator stream -- it is also copyable with copy.copy(). The itertools.count() function is also copyable. Running copy.copy() on other itertools is currently undefined (though I may add copy support to itertools.repeat() and the combinatoric functions). Also, I seems to me the copy.copy() is itself not very bright about what it tries to copy and in giving clear messages about whether or not it successfully made a copy. Raymond From tjreedy at udel.edu Fri Apr 2 01:52:43 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 01 Apr 2010 19:52:43 -0400 Subject: [Python-Dev] copying of itertools iterators In-Reply-To: References: Message-ID: On 4/1/2010 7:20 PM, Andrew Svetlov wrote: > using of copy.copy for simple iterators is forbidden > >>>> import copy >>>> copy.copy(iter([1, 2, 3])) > Traceback (most recent call last): > File "", line 1, in > File "/home/andrew/projects/py3k/Lib/copy.py", line 96, in copy > return _reconstruct(x, rv, 0) > File "/home/andrew/projects/py3k/Lib/copy.py", line 284, in _reconstruct > y = callable(*args) > File "/home/andrew/projects/py3k/Lib/copyreg.py", line 88, in __newobj__ > return cls.__new__(cls, *args) > TypeError: object.__new__(list_iterator) is not safe, use > list_iterator.__new__() The same happens for the iterators of other builtin classes: tuples, sets, and dicts (that I tried). In the other hand, >>> copy.copy(iter(range(1,3,1))) # 3.1 Traceback (most recent call last): ... TypeError: rangeiter() requires 3 int arguments and similar for filter and map. I do not know whether the former group is detected by rule or explicit hard-coded list, but I suspect the latter. > That behavior is safe and clean. > But it's possible to copy iterator objects returned by itertools functions: > >>>> i = itertools.chain([1, 2], [3, 4, 5]) >>>> i.__next__() > 1 >>>> j = copy.copy(i) This works because itertools.chain() is legal >>>> j.__next__() > Traceback (most recent call last): > File "", line 1, in > StopIteration Because itertools.chain() is empty. >>>> i.__next__() > 2 > > Looks like itertools object should be protected from usage like that. I suspect only those for which itertools.xxx() works rather than raising an exception. > Folks, what are you think about? Why privilige the itertools module? A possible rule would be to not copy anything with both .__iter__ and .__next__. Terry Jan Reedy From glyph at twistedmatrix.com Fri Apr 2 01:56:43 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 1 Apr 2010 19:56:43 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: Message-ID: First: thank you distutils-sig, and especially Tarek, for spearheading this effort! I'm particularly excited about the "Distribution" object that this PEP specifies. I've been waiting for a long time to be able to load an object describing a distribution, rather than running setup.py and hoping that it mutated the right state! On Apr 1, 2010, at 5:51 PM, Tarek Ziad? wrote: > - to provide a basic *uninstaller* feature in the distutils2 project. Second: It seems to me that a major missing feature in the PEP is the ability to run some code during installation and uninstallation, especially since it is so easy to run ad-hoc code in setup.py with no way of un-doing what it did. Twisted's plugin system needs this in order to update a plugin index so that plugins can be quickly scanned without being loaded. However, since this is arguably a design flaw in Twisted that should be fixed, I should point out there are other systems that have similar requirements, which are considerably less mutable: COM registration, other registry keys, adding / removing crontab entries, windows services, start menu items, XDG desktop / menu entries, login items, edits to the user's shell configuration, etc. The list goes on and on. I appreciate the "installer marker" feature, since that will at least allow easy_install or pip or something like them to implement this feature with minimal risk of being broken by built-in package management tools, but it seems like such a simple addition that it would be a shame to leave it out. If we could get rid of setup.py entirely so that it wasn't so easy to run ad-hoc stuff during install, I would be happy to leave it to them :). I realize that there are a lot of potential complexities that might creep into the process of determining the execution environment for the code in question, but I personally think it would be good enough to say "You'd better be darn sure to encode all of the run-time state that you need into your own script, or it might break." Third: The PEP is silent on what happens to files whose hash _has_ changed from its install-time value. I guess the implied plan would be to leave them in place. However, this may have nasty side-effects; for example, if the files are test files, then they might be loaded during test discovery, and report exceptions since the code that they're testing has been removed. My suggestion would be to have a specific "quarantine" area where the distutils uninstaller can put modified files that would have been removed as part of a specific distribution, so they aren't still present on PYTHONPATH. I can also think of reasons why you might not want to do this, but either way, the consequence of changing an installed file should be made explicitly clear in the PEP: if they are to be left in place, it should emphasize that point. Finally, one minor bit of bikeshedding, of which I promise to say nothing more if there is not unanimous agreement: I dislike the use of "get_" in function names, since it adds more characters without really adding more information. get_file_users is particularly bad, since it makes me think that it's going to return a list of processes with a file open, or a list of UIDs or something like that. I suggest these names instead: get_distributions() -> active_distributions() get_distribution(name) -> active_distribution_named(name) get_file_users(path) -> distributions_using_path(path) where "active" means "on the current sys.path and thereby accessible by 'import'". This naming would also make the behavior of get_file_users a bit clearer; if the intention is to return only active, loadable distributions (you don't want to be able to use get_file_users to inspect other Python installations or virtualenvs) it could be called active_distributions_using_path. Thanks again to the PEP's author and many contributors, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Fri Apr 2 11:38:02 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 2 Apr 2010 10:38:02 +0100 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> Message-ID: On Mon, Mar 22, 2010 at 7:52 PM, Guido van Rossum wrote: > On Mon, Mar 22, 2010 at 11:36 AM, Raymond Hettinger > wrote: >> One other thought. >> >> The Decimal constructor should now accept floats as a possible input type. >> Formerly, we separated that out to Decimal.from_float() because >> decimals weren't interoperable with floats. > > Not sure this follows; Fraction(1.1) raises an exception, you have to > use Fraction.from_float(). Is there any good reason for this, other than a parallel with Decimal? It seems to me that Raymond's arguments for allowing direct construction of a Decimal from a float apply equally well to the Fraction type. If we're going to allow Decimal(1.1), I'd like to allow Fraction(1.1) to succeed as well (giving the equivalent of Fraction.from_float(1.1)). The main argument against allowing this (for both Fraction and Decimal) seems to be that the result of Decimal(1.1) or Fraction(1.1) could be confusing. But it's an immediate, explicit confusion, which can be quickly resolved by pointing the confusee to the section on floating-point in the appendix, so I don't find this objection particularly compelling. Mark From techtonik at gmail.com Fri Apr 2 15:05:28 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 2 Apr 2010 16:05:28 +0300 Subject: [Python-Dev] 5 : 1 Message-ID: I reviewed 5 issues and want to see http://bugs.python.org/issue7585 committed to Python 2.7 http://bugs.python.org/issue7443 test.support.unlink issue on Windows platform i race condition between os.unlink(), TortoiseSVN and os.open() - reporter changed OS and can't confirm bug anymore, need somebody with Windows and TortoiseSVN installed http://bugs.python.org/issue6703 cross platform failure and silly test in doctest i non-crossplatform check for absolute path when 'module_relative' is True - added patch - need testcases http://bugs.python.org/issue1659 Tests needing network flag? i skip tests that require network if network is not available - added patch - it is recommended to split issue to count skipped tests exactly and to probe which tests marked as 'require network' are actually don't http://bugs.python.org/issue2810 _winreg.EnumValue fails when the registry data includes multibyte unicode characters i WindowsError: [Error 234] More data is available when working with _winreg - original issue assumed that it is caused by multibyte string, however this can not be confirmed anymore - exception can be thrown when operating over long keys, this should be tested with WinAPI over ctypes, I propose to open new bug for long keys - another bug could be opened - it seems impossible to get values of Unicode keys with _winreg http://bugs.python.org/issue3778 python uninstaller leave registry entries - confirmed for 2.6.5 - need windows installer expert to create patch for cleaning empty registry keys -- anatoly t. From stefan at bytereef.org Fri Apr 2 16:54:08 2010 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 2 Apr 2010 16:54:08 +0200 Subject: [Python-Dev] [buildbots] 'stop build' button causing subsequent builds to fail? Message-ID: <20100402145408.GA21742@yoda.bytereef.org> Hi, I looks like the 'stop build' button can a) cause subsequent builds to fail and b) cause pending builds to be deleted from the queue. a) http://www.python.org/dev/buildbot/builders/ARM Linux EABI 3.x/builds/18 was apparently interrupted by a 'stop build' for a previous build. b) I stopped http://www.python.org/dev/buildbot/builders/sparc solaris10 gcc 3.x/builds/558 and a pending build vanished (I'm certain that I used 'stop build' and not 'cancel all'). Is this a known issue? Stefan Krah From dickinsm at gmail.com Fri Apr 2 17:14:14 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 2 Apr 2010 16:14:14 +0100 Subject: [Python-Dev] [buildbots] 'stop build' button causing subsequent builds to fail? In-Reply-To: <20100402145408.GA21742@yoda.bytereef.org> References: <20100402145408.GA21742@yoda.bytereef.org> Message-ID: On Fri, Apr 2, 2010 at 3:54 PM, Stefan Krah wrote: > > I looks like the 'stop build' button can a) cause subsequent builds to fail > and b) cause pending builds to be deleted from the queue. > > > a) http://www.python.org/dev/buildbot/builders/ARM Linux EABI 3.x/builds/18 > ? was apparently interrupted by a 'stop build' for a previous build. Actually, I think that was me being impatient. I was trying to get some information about the float.fromhex test failure on ARM (bugs.python.org/issue8265) and didn't want to wait days. :) > b) I stopped http://www.python.org/dev/buildbot/builders/sparc solaris10 gcc 3.x/builds/558 > ? and a pending build vanished (I'm certain that I used 'stop build' and not 'cancel all'). Don't know about this one. Mark From status at bugs.python.org Fri Apr 2 18:07:38 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 2 Apr 2010 18:07:38 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100402160738.72956780D5@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-03-26 - 2010-04-02) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2629 open (+37) / 17481 closed (+13) / 20110 total (+50) Open issues with patches: 1063 Average duration of open issues: 727 days. Median duration of open issues: 486 days. Open Issues Breakdown open 2590 (+37) languishing 5 ( +0) pending 33 ( +0) Issues Created Or Reopened (51) _______________________________ multiprocessing.Queue() blocks program 2010-03-26 http://bugs.python.org/issue8237 reopened eua py2_test_grammar.py contains invalid syntax for 2.6 2010-03-27 http://bugs.python.org/issue8241 created gagenellina Support surrogates in import ; install Python in a non-ASCII d 2010-03-27 http://bugs.python.org/issue8242 created haypo patch curses writing to window's bottom right position raises: `_cur 2010-03-27 http://bugs.python.org/issue8243 created theosp Remove obsolete update.sh in PEPs repo 2010-03-27 CLOSED http://bugs.python.org/issue8244 created merwok email examples don't actually work (SMTP.connect is not called 2010-03-27 http://bugs.python.org/issue8245 created jaraco test_signal in test_subprocess displays traceback 2010-03-27 CLOSED http://bugs.python.org/issue8246 created pitrou Can't Import Tkinter 2010-03-28 http://bugs.python.org/issue8247 created SevenThunders Add test cases for bool 2010-03-28 CLOSED http://bugs.python.org/issue8248 created gnofi patch expat.ParserCreate - SystemError bad argument to internal func 2010-03-28 http://bugs.python.org/issue8249 created flox Implement pkgutil APIs as described in PEP 376 2010-03-28 http://bugs.python.org/issue8250 created tarek WeakRefSet 2010-03-28 CLOSED http://bugs.python.org/issue8251 created michael.foord add a metadata section in setup.cfg 2010-03-28 http://bugs.python.org/issue8252 created tarek add a resource+files section in setup.cfg 2010-03-28 http://bugs.python.org/issue8253 created tarek write a configure command 2010-03-28 http://bugs.python.org/issue8254 created tarek step-by-step tutorial 2010-03-28 http://bugs.python.org/issue8255 created tarek TypeError: bad argument type for built-in operation 2010-03-28 http://bugs.python.org/issue8256 created dangyogi patch Decimal constructor to accept float 2010-03-28 http://bugs.python.org/issue8257 created rhettinger patch, easy Multiple Python Interpreter Memory Leak 2010-03-29 http://bugs.python.org/issue8258 created ewillie007 left shift operator doesn't accepts long as second argument 2010-03-29 http://bugs.python.org/issue8259 created owirj When I use codecs.open(...) and f.readline() follow up by f.re 2010-03-29 http://bugs.python.org/issue8260 created harobed patch License link for Python 2.6.5 release is broken 2010-03-29 CLOSED http://bugs.python.org/issue8261 created silverjam bad wording in error message attempting to start a Thread twic 2010-03-30 http://bugs.python.org/issue8262 created gagenellina patch regrtest stops prematurately on FreeBSD buildbot, with "succes 2010-03-30 http://bugs.python.org/issue8263 created flox patch, buildbot hasattr doensn't show private (double underscore) attributes e 2010-03-30 http://bugs.python.org/issue8264 created ncw test_float fails on ARM Linux EABI with soft floating point 2010-03-30 http://bugs.python.org/issue8265 created flox buildbot tarfile library should support xz compression 2010-03-30 CLOSED http://bugs.python.org/issue8266 created thedjatclubrock Tutorial section on dictionary keys recommends sort instead of 2010-03-30 http://bugs.python.org/issue8267 created kent patch Make old-style classes weak referenceable 2010-03-30 CLOSED http://bugs.python.org/issue8268 created pitrou patch Missing return values for PyUnicode C/API functions 2010-03-30 http://bugs.python.org/issue8269 created arnau Should socket.PF_PACKET be removed, in favor of socket.AF_PACK 2010-03-31 http://bugs.python.org/issue8270 created denilsonsa str.decode('utf8', 'replace') -- conformance with Unicode 5.2. 2010-03-31 http://bugs.python.org/issue8271 created sjmachin patch Odd exception messages when using cStringIO.StringIO instances 2010-03-31 CLOSED http://bugs.python.org/issue8272 created workshed move test_support into the unittest package 2010-03-31 http://bugs.python.org/issue8273 created tarek test_run failing 2010-03-31 http://bugs.python.org/issue8274 created zubin71 callback function on win64 results in bad behavior. mem corrup 2010-03-31 http://bugs.python.org/issue8275 created ocrush useless PyEval_CallObject function 2010-03-31 CLOSED http://bugs.python.org/issue8276 created fijal ElementTree won't parse comments 2010-04-01 http://bugs.python.org/issue8277 created poke os.utime doesn't allow a atime (Last Access) which is 27 years 2010-04-01 http://bugs.python.org/issue8278 created ramson patch python-gdb PyListTests fail 2010-04-01 http://bugs.python.org/issue8279 created loewis patch urllib2 passes fragment identifier to server 2010-04-01 http://bugs.python.org/issue8280 created naoki test_gdb_sample fails 2010-04-01 CLOSED http://bugs.python.org/issue8281 created pitrou buildbot Windows uninstaller requests admin access for unindentified pr 2010-04-01 CLOSED http://bugs.python.org/issue8282 created techtonik series of lamdas in loop sets the paramater on all calls to th 2010-04-01 CLOSED http://bugs.python.org/issue8283 created alan.hoover urlparse incorrect parse 2010-04-01 CLOSED http://bugs.python.org/issue8284 created OMFGROFLMAO IDLE not smart indenting correctly in nested statements 2010-04-01 http://bugs.python.org/issue8285 created Tofystedeth distutils: path '[...]' cannot end with '/' 2010-04-02 http://bugs.python.org/issue8286 created srid python-gdb.py triggers compile errors on FreeBSD and Solaris 2010-04-02 http://bugs.python.org/issue8287 created flox patch, buildbot zipfile module documentation misprint 2010-04-02 http://bugs.python.org/issue8288 created matpuk multiprocessing.Process.__init__ pickles all arguments 2010-04-02 http://bugs.python.org/issue8289 created cool-RR I HAVE A PROBLEM WITH PYTHON 2010-04-02 http://bugs.python.org/issue8290 created krishnalolla Issues Now Closed (42) ______________________ tarfile insecure pathname extraction 947 days http://bugs.python.org/issue1044 doko patch Remove cmp parameter to list.sort() and builtin.sorted() 813 days http://bugs.python.org/issue1771 r.david.murray os.fstat and other os.f* methods should use PyObject_AsFileDes 697 days http://bugs.python.org/issue2768 georg.brandl patch, easy inspect.getcallargs() 650 days http://bugs.python.org/issue3135 benjamin.peterson patch py3k shouldn't use -fno-strict-aliasing anymore 629 days http://bugs.python.org/issue3326 benjamin.peterson patch ElementPath.Path.findall problem with unicode input 620 days http://bugs.python.org/issue3409 flox patch distutils: -3 warnings (apply) 480 days http://bugs.python.org/issue4577 flox patch Issue with RotatingFileHandler logging handler on Windows 292 days http://bugs.python.org/issue4749 vinay.sajip subprocess.Popen.communicate does not encode unicode strings 408 days http://bugs.python.org/issue5290 flox future unicode literals and r'\u' 389 days http://bugs.python.org/issue5447 flox patch Windows install error when choosing to compile .py files 221 days http://bugs.python.org/issue6716 gagenellina Tkinter sets an unicode environment variable on win32 195 days http://bugs.python.org/issue6906 flox patch, needs review, buildbot What is a Unicode line break character? 83 days http://bugs.python.org/issue7643 flox patch Revise generator-related Glossary entries 37 days http://bugs.python.org/issue8012 georg.brandl patch Add gdb7 hooks to make it easier to debug Python 32 days http://bugs.python.org/issue8032 doko patch Fail-fast behavior for unittest 22 days http://bugs.python.org/issue8074 michael.foord urllib.unquote doesn't decode mixed-case percent escapes 15 days http://bugs.python.org/issue8135 orsenthil patch test_pep277 fails on OS X 5 days http://bugs.python.org/issue8207 flox patch OptionParser keyword arg 'epilog' not mentioned in the docs 6 days http://bugs.python.org/issue8209 orsenthil Python 3 ignored PYTHONUNBUFFERED and -u 10 days http://bugs.python.org/issue8213 georg.brandl enabling SSL_MODE_AUTO_RETRY on SSL sockets 2 days http://bugs.python.org/issue8222 pitrou Wrong link in xml.etree documentation 7 days http://bugs.python.org/issue8225 brian.curtin patch extend py_compile to compile files from stdin 6 days http://bugs.python.org/issue8233 barry patch Spelling and text in howto/webservers 1 days http://bugs.python.org/issue8234 ezio.melotti patch Support FreeBSD's SO_SETFIB in socketmodule.c 8 days http://bugs.python.org/issue8235 larry patch ./configure: ImportError: No module named asdl (when run from 4 days http://bugs.python.org/issue8236 r.david.murray Remove obsolete update.sh in PEPs repo 0 days http://bugs.python.org/issue8244 pitrou test_signal in test_subprocess displays traceback 0 days http://bugs.python.org/issue8246 flox Add test cases for bool 3 days http://bugs.python.org/issue8248 pitrou patch WeakRefSet 0 days http://bugs.python.org/issue8251 michael.foord License link for Python 2.6.5 release is broken 4 days http://bugs.python.org/issue8261 georg.brandl tarfile library should support xz compression 0 days http://bugs.python.org/issue8266 brian.curtin Make old-style classes weak referenceable 1 days http://bugs.python.org/issue8268 pitrou patch Odd exception messages when using cStringIO.StringIO instances 0 days http://bugs.python.org/issue8272 eric.smith useless PyEval_CallObject function 1 days http://bugs.python.org/issue8276 pitrou test_gdb_sample fails 0 days http://bugs.python.org/issue8281 pitrou buildbot Windows uninstaller requests admin access for unindentified pr 0 days http://bugs.python.org/issue8282 loewis series of lamdas in loop sets the paramater on all calls to th 0 days http://bugs.python.org/issue8283 benjamin.peterson urlparse incorrect parse 0 days http://bugs.python.org/issue8284 georg.brandl Generate from Unicode database instead of manualy coding. 1400 days http://bugs.python.org/issue1498930 flox patch str.split creates new string even if pattern not found 1207 days http://bugs.python.org/issue1613130 flox object.__init__ shouldn't allow args/kwds 1110 days http://bugs.python.org/issue1683368 tjreedy Top Issues Most Discussed (10) ______________________________ 23 test_zlib fails with zlib 1.2.4 12 days open http://bugs.python.org/issue8193 20 str.decode('utf8', 'replace') -- conformance with Unicode 5.2.0 3 days open http://bugs.python.org/issue8271 17 move test_support into the unittest package 2 days open http://bugs.python.org/issue8273 15 Fix typos and phrasing in the Web servers howto 10 days open http://bugs.python.org/issue8218 13 ssl.SSLSocket.write may fail on non-blocking sockets 7 days open http://bugs.python.org/issue8240 11 ABC caches should use weak refs 732 days open http://bugs.python.org/issue2521 11 os.kill on windows 1753 days open http://bugs.python.org/issue1220212 8 TypeError: bad argument type for built-in operation 5 days open http://bugs.python.org/issue8256 8 python uninstaller leave registry entries 575 days open http://bugs.python.org/issue3778 6 urlparse incorrect parse 0 days closed http://bugs.python.org/issue8284 From guido at python.org Fri Apr 2 19:10:43 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Apr 2010 10:10:43 -0700 Subject: [Python-Dev] Mixing float and Decimal -- thread reboot In-Reply-To: References: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com> <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com> <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com> Message-ID: On Fri, Apr 2, 2010 at 2:38 AM, Mark Dickinson wrote: > On Mon, Mar 22, 2010 at 7:52 PM, Guido van Rossum wrote: >> On Mon, Mar 22, 2010 at 11:36 AM, Raymond Hettinger >> wrote: >>> One other thought. >>> >>> The Decimal constructor should now accept floats as a possible input type. >>> Formerly, we separated that out to Decimal.from_float() because >>> decimals weren't interoperable with floats. >> >> Not sure this follows; Fraction(1.1) raises an exception, you have to >> use Fraction.from_float(). > > Is there any good reason for this, other than a parallel with Decimal? > ?It seems to me that Raymond's arguments for allowing direct > construction of a Decimal from a float apply equally well to the > Fraction type. > > If we're going to allow Decimal(1.1), I'd like to allow Fraction(1.1) > to succeed as well (giving the equivalent of > Fraction.from_float(1.1)). > > The main argument against allowing this (for both Fraction and > Decimal) seems to be that the result of Decimal(1.1) or Fraction(1.1) > could be confusing. ?But it's an immediate, explicit confusion, which > can be quickly resolved by pointing the confusee to the section on > floating-point in the appendix, so I don't find this objection > particularly compelling. Agreed. If people don't learn about from_float() they might do weird stuff like Decimal(0) + 1.1. Making Fraction and Decimal do the same thing should be easy enough. -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Fri Apr 2 19:59:04 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 02 Apr 2010 13:59:04 -0400 Subject: [Python-Dev] 5 : 1 In-Reply-To: References: Message-ID: <20100402175904.908BF1BC331@kimball.webabinitio.net> On Fri, 02 Apr 2010 16:05:28 +0300, anatoly techtonik wrote: > I reviewed 5 issues and want to see http://bugs.python.org/issue7585 > committed to Python 2.7 It may be a little too close to the feature freeze to get this reviewed and in to 2.7. I'll take a look at it tonight (I'm in GMT -4) if I can (and I think I can). -- R. David Murray www.bitdance.com From solipsis at pitrou.net Fri Apr 2 20:06:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Apr 2010 18:06:28 +0000 (UTC) Subject: [Python-Dev] 5 : 1 References: <20100402175904.908BF1BC331@kimball.webabinitio.net> Message-ID: R. David Murray bitdance.com> writes: > > On Fri, 02 Apr 2010 16:05:28 +0300, anatoly techtonik gmail.com> wrote: > > I reviewed 5 issues and want to see http://bugs.python.org/issue7585 > > committed to Python 2.7 > > It may be a little too close to the feature freeze to get this reviewed > and in to 2.7. This looks like a bug fix, not a new feature. Regards Antoine. From stefan at bytereef.org Fri Apr 2 20:38:25 2010 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 2 Apr 2010 20:38:25 +0200 Subject: [Python-Dev] [buildbots] 'stop build' button causing subsequent builds to fail? In-Reply-To: References: <20100402145408.GA21742@yoda.bytereef.org> Message-ID: <20100402183825.GB22963@yoda.bytereef.org> Mark Dickinson wrote: > On Fri, Apr 2, 2010 at 3:54 PM, Stefan Krah wrote: > > > > I looks like the 'stop build' button can a) cause subsequent builds to fail > > and b) cause pending builds to be deleted from the queue. > > > > > > a) http://www.python.org/dev/buildbot/builders/ARM Linux EABI 3.x/builds/18 > > ? was apparently interrupted by a 'stop build' for a previous build. > > Actually, I think that was me being impatient. I was trying to get > some information about the float.fromhex test failure on ARM > (bugs.python.org/issue8265) and didn't want to wait days. :) > > > b) I stopped http://www.python.org/dev/buildbot/builders/sparc solaris10 gcc 3.x/builds/558 > > ? and a pending build vanished (I'm certain that I used 'stop build' and not 'cancel all'). > > Don't know about this one. I was misled by the fact that the http://www.python.org/dev/buildbot/builders/XYZ pages often show 'no current builds' even when there is a build going on. The http://www.python.org/dev/buildbot/buildslaves/XYZ pages appear to give accurate information. Stefan Krah From rdmurray at bitdance.com Fri Apr 2 20:50:42 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 02 Apr 2010 14:50:42 -0400 Subject: [Python-Dev] 5 : 1 In-Reply-To: References: <20100402175904.908BF1BC331@kimball.webabinitio.net> Message-ID: <20100402185042.88F8D1FC1F8@kimball.webabinitio.net> On Fri, 02 Apr 2010 18:06:28 -0000, Antoine Pitrou wrote: > R. David Murray bitdance.com> writes: > > > > On Fri, 02 Apr 2010 16:05:28 +0300, anatoly techtonik > gmail.com> wrote: > > > I reviewed 5 issues and want to see http://bugs.python.org/issue7585 > > > committed to Python 2.7 > > > > It may be a little too close to the feature freeze to get this reviewed > > and in to 2.7. > > This looks like a bug fix, not a new feature. Well, yes, but it is also a behavior change. Is that OK for a Beta? --David From solipsis at pitrou.net Fri Apr 2 20:58:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Apr 2010 18:58:07 +0000 (UTC) Subject: [Python-Dev] 5 : 1 References: <20100402175904.908BF1BC331@kimball.webabinitio.net> <20100402185042.88F8D1FC1F8@kimball.webabinitio.net> Message-ID: R. David Murray bitdance.com> writes: > > On Fri, 02 Apr 2010 18:06:28 -0000, Antoine Pitrou wrote: > > R. David Murray bitdance.com> writes: > > > > > > On Fri, 02 Apr 2010 16:05:28 +0300, anatoly techtonik > > gmail.com> wrote: > > > > I reviewed 5 issues and want to see http://bugs.python.org/issue7585 > > > > committed to Python 2.7 > > > > > > It may be a little too close to the feature freeze to get this reviewed > > > and in to 2.7. > > > > This looks like a bug fix, not a new feature. > > Well, yes, but it is also a behavior change. Is that OK for a Beta? As soon as we consider the old behaviour a bug, I'd say yes. Besides, this is the first beta, things get stricter afterwards. Regards Antoine. From pythonsteve at gmail.com Sat Apr 3 13:16:47 2010 From: pythonsteve at gmail.com (Steve Bonner) Date: Sat, 3 Apr 2010 07:16:47 -0400 Subject: [Python-Dev] nonlocals() function? Message-ID: What do we think of adding a built-in nonlocals() function that would be similar to globals() and locals()? Like those functions, it would return a dictionary of variable names and their values. Since we now have the nonlocal statement, it would be consistent to keep the three scopes local/nonlocal/global with parallel capabilities. And it might sometimes be useful for code inside a nested function to see what variables are available at the enclosing level. Steve Bonner -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Apr 3 15:31:59 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Apr 2010 23:31:59 +1000 Subject: [Python-Dev] nonlocals() function? In-Reply-To: References: Message-ID: <4BB7434F.8000107@gmail.com> Steve Bonner wrote: > What do we think of adding a built-in nonlocals() function that would > be similar to globals() and locals()? Like those functions, it would > return a dictionary of variable names and their values. Since we now > have the nonlocal statement, it would be consistent to keep the > three scopes local/nonlocal/global with parallel capabilities. And it > might sometimes be useful for code inside a nested function to see > what variables are available at the enclosing level. That isn't as easy as it may sound. locals() and globals() are each single namespaces, while the nonlocals may actually span multiple namespaces. Python actually deals with this at compilation time and when the function object is created - the interpreter knows the names of all the nonlocal cells that need to be connected, and creates the appropriate links to the cells in the outer scopes. That situation doesn't translate well to dict-style access - while the whole local namespace is readily accessible to the interpreter, as is a pointer to the module namespace (for globals()), no such convenient data source exists for the full set of possible nonlocal references from an arbitrary function. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From benjamin at python.org Sat Apr 3 18:42:40 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Apr 2010 10:42:40 -0600 Subject: [Python-Dev] delaying 2.7 beta 1 Message-ID: I am delaying the release of 2.7 beta 1 pending the fixing of http://bugs.python.org/issue8108 and the greening of the buildbots. -- Regards, Benjamin From benjamin at python.org Sat Apr 3 22:45:57 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Apr 2010 14:45:57 -0600 Subject: [Python-Dev] delaying 2.7 beta 1 In-Reply-To: References: Message-ID: 2010/4/3 Benjamin Peterson : > I am delaying the release of 2.7 beta 1 pending the fixing of > http://bugs.python.org/issue8108 and the greening of the buildbots. After consultation with Antoine I've lowered #8108's priority. I will see if the buildbots shape up... -- Regards, Benjamin From ziade.tarek at gmail.com Sun Apr 4 00:48:56 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 4 Apr 2010 00:48:56 +0200 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: <20100401233652.764823A40AF@sparrow.telecommunity.com> References: <20100401233652.764823A40AF@sparrow.telecommunity.com> Message-ID: 2010/4/2 P.J. Eby : [..] > First, I notice the RECORD paths are listed as being relative to sys.prefix; > I was under the impression that these paths were supposed to be relative to > the base installation location of the library (i.e. site-packages in the > default case). > > That is, that while paths not under sys.prefix were to be absolute, the > paths under sys.prefix were relative to the install location. ?(Since one of > the original motivating use cases for relative paths in RECORD was to > support relocatable installations, and sharing installations across > platforms, when the code is platform independent code.) ?Also, the spec > doesn't address installation to paths that aren't under sys.prefix, or what > happens if you use "setup.py install --prefix". > > I'm wondering if perhaps this should be restated as something like: > > * Paths under the base installation location are relative to the base > * Paths not under the base installation location, but under the installation > prefix, are also stored relative to the base, IF the base location is a > subpath of the installation prefix > * All other paths are absolute. > > Where "base location" is the effective --install-lib directory, and prefix > is the effective --prefix. ?(Which default of course to site-package and > sys.prefix respectively, but the spec shouldn't be in terms of those > defaults.) Sounds like a better solution. > > Second, the RECORD spec is not clear on what happens to the hash field on > .pyc/.pyo files. ?Is it supposed to be an empty string, a hash of an empty > file, a hash value that's ignored, or...? The example misses pyc/pyo files. Those would have no hash, and I also think the size should be omited as well. I'll update the example. > > Third, what is a "local absolute path"? ?This term appears several places > but is not defined. ?It sounds like maybe you mean a path using the local > platform separator, as opposed to a '/'-separated relative path. ?If that's > the case, this should probably be spelled out somewhere early on and then > referenced in context. Yes, this is it. It was defined somewhere but removed at some point by mistake IIRC. I'll add that > > Fourth, the PEP says the implementation will support zipfiles, but the draft > implementation doesn't appear to support zipimport subdirectories. > ?(Zipimport paths can be /path/to/some.zip/subdir, and this base is then > used for importing, just as if the zipfile were a normal parent directory.) > ?If this is an intentional omission, it should probably be mentioned in the > PEP. The implementation so far will load zip files founded in the paths, see ZippedDistributionDir/ZippedDistribution. I am wondering though, if we shouldn't omit this zip story for PEP 376 altogether, and work later on something more generic wrt import protocols where modules/packages could be stored anywhere. IOW, PEP 376 could focus exclusively on a pure filesystem site-packages. > > Okay, on to the minor corrections. ?The following uses of "package" should > be replaced by "project" or "distribution": > > "If a package that was already installed on the system as a dependency" > "in other words, whether the package was installed by user request" > "$ python -m distutils.uninstall packagename" yes, will edit this. > Anyway, that's my first pass through; I think the RECORD section may need > further fleshing out (e.g., specifying a particular csv-module dialect) in > order to ensure that the spec is sufficiently rigorous for multiple tools to > consume/produce compatible files. Yes, and we should list those tools somewhere in the PEP I guess Regards Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Sun Apr 4 01:05:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 4 Apr 2010 01:05:34 +0200 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 1:56 AM, Glyph Lefkowitz wrote: > First: thank you distutils-sig, and especially Tarek, for spearheading this > effort! Thanks :) > I'm particularly excited about the "Distribution" object that this PEP > specifies. ?I've been waiting for a long time to be able to load an object > describing a distribution, rather than running setup.py and hoping that it > mutated the right state! Notice that this Distribution class will work with installed distributions, where the info where built by running setup.py. As opposed to distritbutions that are not yet installed where these infos are still hidden in setup.py. Although we are currently working in distutils2 to get rid of setup.py and have a pure static metadata file. > On Apr 1, 2010, at 5:51 PM, Tarek Ziad? wrote: > > - to provide a basic *uninstaller* feature in the distutils2 project. > > Second: It seems to me that a major missing feature in the PEP is the > ability to run some code during installation and uninstallation, especially > since it is so easy to run ad-hoc code in setup.py with no way of un-doing > what it did. That was a feature request for distutils: install/removal hooks like what RPM or the msi installer offers. I think this could be done in distutils2 no matter what happens to the PEP. I am adding this in the bug tracker. > Third: The PEP is silent on what happens to files whose hash _has_ changed > from its install-time value. ?I guess the implied plan would be to leave > them in place. ?However, this may have nasty side-effects; for example, if > the files are test files, then they might be loaded during test discovery, > and report exceptions since the code that they're testing has been removed. > ?My suggestion would be to have a specific "quarantine" area where the > distutils uninstaller can put modified files that would have been removed as > part of a specific distribution, so they aren't still present on PYTHONPATH. > ?I can also think of reasons why you might not want to do this, but either > way, the consequence of changing an installed file should be made explicitly > clear in the PEP: if they are to be left in place, it should emphasize that > point. The strategy about what should be done with modified files upon uninstallation is up to the implementation I guess. But the PEP should mention what Distutils2 basic uninstall function should do. The quarantine strategy sounds like a great idea. I propose to add this in the PEP in the implementation details section, and say in the specs that it's up to the implementation to decide. > Finally, one minor bit of bikeshedding, of which I promise to say nothing > more if there is not unanimous agreement: I dislike the use of "get_" in > function names, since it adds more characters without really adding more > information. ?get_file_users is particularly bad, since it makes me think > that it's going to return a list of processes with a file open, or a list of > UIDs or something like that. ?I suggest these names instead: > ?? ?get_distributions() -> active_distributions() > ?? ?get_distribution(name) -> active_distribution_named(name) > ?? ?get_file_users(path) -> distributions_using_path(path) > where "active" means "on the current sys.path and thereby accessible by > 'import'". "active" sounds superfluous too I think, since all distributions that are found *are* active. So it could be "distributions()". But I am not comfortable with this single word because it sounds to me like a sequence object rather than a function. But that might be just me.. > This naming would also make the behavior of get_file_users a bit > clearer; if the intention is to return only active, loadable distributions > (you don't want to be able to use get_file_users to inspect other Python > installations or virtualenvs) it could be called > active_distributions_using_path. +1 on distributions_using_path() > Thanks again to the PEP's author and many contributors, Thanks for the feedback ! -- Tarek Ziad? | http://ziade.org From python at mrabarnett.plus.com Sun Apr 4 02:40:29 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Apr 2010 01:40:29 +0100 Subject: [Python-Dev] Odd lines in unicodedata_db.h Message-ID: <4BB7DFFD.6000900@mrabarnett.plus.com> I've just downloaded the daily snapshot at http://svn.python.org/snapshots/python.tar.bz2 In the header file /python/Modules/unicodedata_db.h, there are the following lines in the change_records_3_2_0 struct: { 255, 255, 255, 255, 1.0 }, { 255, 255, 255, 255, 2.0 }, { 255, 255, 255, 255, 3.0 }, { 255, 255, 255, 255, 4.0 }, ... { 255, 255, 255, 255, 1e+16 }, { 255, 255, 255, 255, 1e+20 }, Looks like a bug to me. From amauryfa at gmail.com Sun Apr 4 11:21:49 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sun, 4 Apr 2010 11:21:49 +0200 Subject: [Python-Dev] Odd lines in unicodedata_db.h In-Reply-To: <4BB7DFFD.6000900@mrabarnett.plus.com> References: <4BB7DFFD.6000900@mrabarnett.plus.com> Message-ID: 2010/4/4 MRAB : > I've just downloaded the daily snapshot at > http://svn.python.org/snapshots/python.tar.bz2 > > In the header file /python/Modules/unicodedata_db.h, there are the > following lines in the change_records_3_2_0 struct: > > ? ? ? ?{ 255, 255, 255, 255, 1.0 }, > ? ? ? ?{ 255, 255, 255, 255, 2.0 }, > ? ? ? ?{ 255, 255, 255, 255, 3.0 }, > ? ? ? ?{ 255, 255, 255, 255, 4.0 }, > ? ? ? ?... > ? ? ? ?{ 255, 255, 255, 255, 1e+16 }, > ? ? ? ?{ 255, 255, 255, 255, 1e+20 }, > > Looks like a bug to me. I don't think so. Unicode 3.2 did contain two entries with large numeric values. The file Unihan-3.2.0.txt contains these two lines: U+4EAC kPrimaryNumeric 10,000,000,000,000,000 ten quadrillion (American) U+5793 kPrimaryNumeric 100,000,000,000,000,000,000 hundred quintillion (American) For some reason newer versions of the unicode standard removed these values. -- Amaury Forgeot d'Arc From stephen at xemacs.org Sun Apr 4 12:59:14 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 04 Apr 2010 19:59:14 +0900 Subject: [Python-Dev] Odd lines in unicodedata_db.h In-Reply-To: References: <4BB7DFFD.6000900@mrabarnett.plus.com> Message-ID: <87tyrrpjp9.fsf@uwakimon.sk.tsukuba.ac.jp> Amaury Forgeot d'Arc writes: > I don't think so. Unicode 3.2 did contain two entries with large > numeric values. The file Unihan-3.2.0.txt contains these two > lines: > > U+4EAC kPrimaryNumeric 10,000,000,000,000,000 ten quadrillion (American) > U+5793 kPrimaryNumeric 100,000,000,000,000,000,000 hundred quintillion > (American) They are related to the Chinese numbering system. I recall U+4EAC having that value from my textbooks (it's the "kyo" in Tokyo, and the "jing" in "Beijing", so quite memorable), and U+5793 looks familiar (it's not otherwise used in Japanese AFAIK, so I'm not sure, but it seems quite plausible that there would be a character for 10000^5). > For some reason newer versions of the unicode standard removed > these values. The characters are still there. The numeric values were probably removed because in practice they're not actually used (at least, almost never in Japanese). It seems a little sad to save 150 bytes or so in a table and lose the historical meanings. From solipsis at pitrou.net Sun Apr 4 13:41:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 4 Apr 2010 11:41:29 +0000 (UTC) Subject: [Python-Dev] delaying 2.7 beta 1 References: Message-ID: Le Sat, 03 Apr 2010 14:45:57 -0600, Benjamin Peterson a ?crit?: > 2010/4/3 Benjamin Peterson : >> I am delaying the release of 2.7 beta 1 pending the fixing of >> http://bugs.python.org/issue8108 and the greening of the buildbots. > > After consultation with Antoine I've lowered #8108's priority. I will > see if the buildbots shape up... Of course, the chances we have a release are higher if people don't try to sneak in new features two hours before the tagging and the ensueing feature freeze. Regards Antoine. From python at mrabarnett.plus.com Sun Apr 4 19:43:03 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Apr 2010 18:43:03 +0100 Subject: [Python-Dev] Odd lines in unicodedata_db.h In-Reply-To: References: <4BB7DFFD.6000900@mrabarnett.plus.com> Message-ID: <4BB8CFA7.4010400@mrabarnett.plus.com> Amaury Forgeot d'Arc wrote: > 2010/4/4 MRAB : >> I've just downloaded the daily snapshot at >> http://svn.python.org/snapshots/python.tar.bz2 >> >> In the header file /python/Modules/unicodedata_db.h, there are the >> following lines in the change_records_3_2_0 struct: >> >> { 255, 255, 255, 255, 1.0 }, >> { 255, 255, 255, 255, 2.0 }, >> { 255, 255, 255, 255, 3.0 }, >> { 255, 255, 255, 255, 4.0 }, >> ... >> { 255, 255, 255, 255, 1e+16 }, >> { 255, 255, 255, 255, 1e+20 }, >> >> Looks like a bug to me. > > I don't think so. Unicode 3.2 did contain two entries with large numeric values. > The file Unihan-3.2.0.txt contains these two lines: > > U+4EAC kPrimaryNumeric 10,000,000,000,000,000 ten quadrillion (American) > U+5793 kPrimaryNumeric 100,000,000,000,000,000,000 hundred quintillion > (American) > > For some reason newer versions of the unicode standard removed these values. > It causes a type warning: warning C4244: 'initializing' : conversion from 'double' to 'const int', possible loss of data From dreamingforward at gmail.com Mon Apr 5 01:03:17 2010 From: dreamingforward at gmail.com (average) Date: Sun, 4 Apr 2010 16:03:17 -0700 Subject: [Python-Dev] nonlocals() function? In-Reply-To: References: <4BB7434F.8000107@gmail.com> Message-ID: On Sat, Apr 3, 2010 at 6:31 AM, Nick Coghlan wrote: > Steve Bonner wrote: >> What do we think of adding a built-in nonlocals() function that would >> be similar to globals() and locals()? ?Like those functions, it would >> return a dictionary of variable names and their values. Since we now >> have the nonlocal statement, it would be consistent to keep the >> three scopes local/nonlocal/global with parallel capabilities. And it >> might sometimes be useful for code inside a nested function to see >> what variables are available at the enclosing level. > > That isn't as easy as it may sound. locals() and globals() are each > single namespaces, while the nonlocals may actually span multiple > namespaces. This is probably is way off, but why not use sets. ?It seems that these special "functions" (which actually don't do any transformations) should be properties or attributes, and in any case would be better served by returning (holding) sets, then one could do standard set operations on them (including the set difference operations upon the set named "locals"). These old functions predate the introductions of attributes/properties and sets, and like anything that is sure to be a group containing no duplicates, it should really return a set to remove the ambiguity to the programmer (unlike returning a list as is presently). ?That includes dir() too and probably others. python3k-to-you-toly yours, marcos From guido at python.org Mon Apr 5 01:03:20 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 4 Apr 2010 16:03:20 -0700 Subject: [Python-Dev] Odd lines in unicodedata_db.h In-Reply-To: <4BB8CFA7.4010400@mrabarnett.plus.com> References: <4BB7DFFD.6000900@mrabarnett.plus.com> <4BB8CFA7.4010400@mrabarnett.plus.com> Message-ID: On Sun, Apr 4, 2010 at 10:43 AM, MRAB wrote: > Amaury Forgeot d'Arc wrote: >> >> 2010/4/4 MRAB : >>> >>> I've just downloaded the daily snapshot at >>> http://svn.python.org/snapshots/python.tar.bz2 >>> >>> In the header file /python/Modules/unicodedata_db.h, there are the >>> following lines in the change_records_3_2_0 struct: >>> >>> ? ? ? { 255, 255, 255, 255, 1.0 }, >>> ? ? ? { 255, 255, 255, 255, 2.0 }, >>> ? ? ? { 255, 255, 255, 255, 3.0 }, >>> ? ? ? { 255, 255, 255, 255, 4.0 }, >>> ? ? ? ... >>> ? ? ? { 255, 255, 255, 255, 1e+16 }, >>> ? ? ? { 255, 255, 255, 255, 1e+20 }, >>> >>> Looks like a bug to me. >> >> I don't think so. Unicode 3.2 did contain two entries with large numeric >> values. >> The file Unihan-3.2.0.txt contains these two lines: >> >> U+4EAC ?kPrimaryNumeric 10,000,000,000,000,000 ten quadrillion (American) >> U+5793 ?kPrimaryNumeric 100,000,000,000,000,000,000 hundred quintillion >> (American) >> >> For some reason newer versions of the unicode standard removed these >> values. >> > It causes a type warning: > > warning C4244: 'initializing' : conversion from 'double' to 'const int', > possible loss of data The numeric_changed field isn't used anywhere in the code AFAIK, so I don't see why we need to preserve them. -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Mon Apr 5 01:44:57 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Apr 2010 10:44:57 +1100 Subject: [Python-Dev] nonlocals() function? In-Reply-To: References: Message-ID: <201004050944.58416.steve@pearwood.info> On Mon, 5 Apr 2010 09:03:17 am average wrote: > On Sat, Apr 3, 2010 at 6:31 AM, Nick Coghlan wrote: > > Steve Bonner wrote: > >> What do we think of adding a built-in nonlocals() function that > >> would be similar to globals() and locals()? ?Like those functions, > >> it would return a dictionary of variable names and their values. > >> Since we now have the nonlocal statement, it would be consistent > >> to keep the three scopes local/nonlocal/global with parallel > >> capabilities. And it might sometimes be useful for code inside a > >> nested function to see what variables are available at the > >> enclosing level. > > > > That isn't as easy as it may sound. locals() and globals() are each > > single namespaces, while the nonlocals may actually span multiple > > namespaces. > > This is probably is way off, but why not use sets. ?It seems that > these special "functions" (which actually don't do any > transformations) should be properties or attributes, and in any case > would be better served by returning (holding) sets, then one could do > standard set operations on them (including the set difference > operations upon the set named "locals"). globals() and locals() return dicts mapping names to objects. Returning sets would break code that expects a dict, as well as reducing functionality: they could return the object names, but not the objects themselves. That makes it unobvious how to get to the objects, given the name. (Using eval perhaps?) I'm not sure why you think that the globals() and locals() functions aren't functions. They transform the empty argument into dicts. You say they should be properties and attributes, but properties of what? > These old functions predate the introductions of > attributes/properties and sets, and like anything that is sure to be > a group containing no duplicates, it should really return a set to > remove the ambiguity to the programmer (unlike returning a list as is > presently). ?That includes dir() too and probably others. You are confused -- globals() and locals() don't return lists. Bringing this back to the original question, I very rarely use globals() and even more rarely use locals(), so I don't have much need for a nonlocals() function. I suspect it's a solution in search of a problem, but the same holds for locals() (in my option) and so I wouldn't object if somebody else volunteered to do the work :) +0 -- Steven D'Aprano From cmjohnson.mailinglist at gmail.com Mon Apr 5 05:21:44 2010 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Sun, 4 Apr 2010 17:21:44 -1000 Subject: [Python-Dev] nonlocals() function? In-Reply-To: <201004050944.58416.steve@pearwood.info> References: <201004050944.58416.steve@pearwood.info> Message-ID: On Sun, Apr 4, 2010 at 1:44 PM, Steven D'Aprano wrote: > On Mon, 5 Apr 2010 09:03:17 am average wrote: >> presently). ?That includes dir() too and probably others. > > You are confused -- globals() and locals() don't return lists. Yeah, this thread is mostly a mess (what would nonlocals() return anyway?), but there is one serious suggestion in there. Maybe dir() should return a set. Points in its favor: * The results of dir() are non-repeating * The order of items returned by dir() is not especially important * It would make "method" in dir(obj) marginally faster Points against: * It would break code that tried dir(obj)[0] (but why would anyone do that?) * Even though the order isn?t important for code, it?s convenient at the interactive prompt to see the methods of an item in alphabetical order for quick scanning. Overall: -0 My two bits. -- Carl Johnson From cmjohnson.mailinglist at gmail.com Mon Apr 5 05:34:18 2010 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Sun, 4 Apr 2010 17:34:18 -1000 Subject: [Python-Dev] Odd lines in unicodedata_db.h In-Reply-To: <87tyrrpjp9.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BB7DFFD.6000900@mrabarnett.plus.com> <87tyrrpjp9.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sun, Apr 4, 2010 at 12:59 AM, Stephen J. Turnbull wrote: > They are related to the Chinese numbering system. ?I recall U+4EAC > having that value from my textbooks (it's the "kyo" in Tokyo, and the > "jing" in "Beijing", so quite memorable), and U+5793 looks familiar > (it's not otherwise used in Japanese AFAIK, so I'm not sure, but it > seems quite plausible that there would be a character for 10000^5). For reference, the Japanese dictionary built into OS X defines U+5793 as: Gai Numerical unit. 10,000 ?kei?s (kei = U+4EAC). 10 to the 20th power. Formerly also used for 10 ?kei?s. Reference-deskily-yours, -- Carl Johnson From greg.ewing at canterbury.ac.nz Mon Apr 5 10:32:41 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 05 Apr 2010 20:32:41 +1200 Subject: [Python-Dev] nonlocals() function? In-Reply-To: References: <201004050944.58416.steve@pearwood.info> Message-ID: <4BB9A029.90206@canterbury.ac.nz> Carl M. Johnson wrote: > * It would make "method" in dir(obj) marginally faster Wouldn't hasattr(obj, "method") be a better way to do that? > * Even though the order isn?t important for code, it?s convenient at > the interactive prompt to see the methods of an item in alphabetical > order for quick scanning. Since I suspect this is most people's main use for dir(), I think it's a good enough reason for leaving things as they are. -- Greg From solipsis at pitrou.net Mon Apr 5 14:17:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Apr 2010 12:17:27 +0000 (UTC) Subject: [Python-Dev] nonlocals() function? References: Message-ID: Steve Bonner gmail.com> writes: > > What do we think of adding a built-in nonlocals() function that would > be similar to globals() and locals()? ?Like those functions, it would > return a dictionary of variable names and their values. Since we now > have the nonlocal statement, it would be consistent to keep the > three scopes local/nonlocal/global with parallel capabilities. These scopes don't have parallel capabilities: >>> def f(): ... x = 5 ... locals()['x'] = 6 ... return x ... >>> f() 5 > And it > might sometimes be useful for code inside a nested function to see > what variables are available at the enclosing level. "It might sometimes be useful" translates in my head to "I've never seen an actual use case for this". -1 on an useless complication of the interpreter. From ncoghlan at gmail.com Mon Apr 5 15:03:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 05 Apr 2010 23:03:39 +1000 Subject: [Python-Dev] nonlocals() function? In-Reply-To: <4BB9A029.90206@canterbury.ac.nz> References: <201004050944.58416.steve@pearwood.info> <4BB9A029.90206@canterbury.ac.nz> Message-ID: <4BB9DFAB.2070800@gmail.com> Greg Ewing wrote: >> * Even though the order isn?t important for code, it?s convenient at >> the interactive prompt to see the methods of an item in alphabetical >> order for quick scanning. > > Since I suspect this is most people's main use for > dir(), I think it's a good enough reason for leaving > things as they are. Yes, I was going to point out that I would personally be *very* annoyed if someone removed the automatic sorting from the output of dir() :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From willian at ufpa.br Mon Apr 5 16:54:51 2010 From: willian at ufpa.br (willian at ufpa.br) Date: Mon, 5 Apr 2010 14:54:51 +0000 (UTC) Subject: [Python-Dev] python compiler Message-ID: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> for a college project, I proposed to create a compiler for python. I've read something about it and maybe I saw that made a bad choice. I hear everyone's opinion respond. From fuzzyman at voidspace.org.uk Mon Apr 5 17:11:40 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 05 Apr 2010 16:11:40 +0100 Subject: [Python-Dev] python compiler In-Reply-To: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: <4BB9FDAC.10205@voidspace.org.uk> On 05/04/2010 15:54, willian at ufpa.br wrote: > for a college project, I proposed to create a compiler for python. I've > read something about it and maybe I saw that made a bad choice. I hear > everyone's opinion respond. > Python itself is a highly dynamic language and not amenable to direct compilation. Instead modern just-in-time compiler technology is seen as the way to improve Python performance. Projects that are doing this are PyPy and Unladen Swallow. A static subset of Python can be statically compiled, projects that do that include RPython (part of PyPy) and ShedSkin. These are not really Python though, just Python like languages that happen to be valid subsets of Python. All the best, Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From phd at phd.pp.ru Mon Apr 5 17:13:05 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 5 Apr 2010 19:13:05 +0400 Subject: [Python-Dev] python compiler In-Reply-To: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: <20100405151305.GA8770@phd.pp.ru> Hello. We'are sorry but we cannot help you. This mailing list is to work on developing Python (fixing bugs and adding new features to Python itself); if you're having problems using Python, please find another forum. Probably python-list (comp.lang.python) news group/mailing list is the best place. See http://www.python.org/community/lists/ for other lists/news groups/fora. On Mon, Apr 05, 2010 at 02:54:51PM +0000, willian at ufpa.br wrote: Thank you for understanding. > for a college project, I proposed to create a compiler for python. I've > read something about it and maybe I saw that made a bad choice. I hear > everyone's opinion respond. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From rnk at mit.edu Mon Apr 5 17:25:36 2010 From: rnk at mit.edu (Reid Kleckner) Date: Mon, 5 Apr 2010 11:25:36 -0400 Subject: [Python-Dev] python compiler In-Reply-To: <4BB9FDAC.10205@voidspace.org.uk> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: On Mon, Apr 5, 2010 at 11:11 AM, Michael Foord wrote: > Python itself is a highly dynamic language and not amenable to direct > compilation. Instead modern just-in-time compiler technology is seen as the > way to improve Python performance. Projects that are doing this are PyPy and > Unladen Swallow. A static subset of Python can be statically compiled, > projects that do that include RPython (part of PyPy) and ShedSkin. These are > not really Python though, just Python like languages that happen to be valid > subsets of Python. I agree. However, if you're doing it as a fun final project and don't care about performance and don't mind generating slow code, then go for it. You'd also want to cut a bunch of corners like exec and eval. Reid From craigcitro at gmail.com Mon Apr 5 17:31:54 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 5 Apr 2010 08:31:54 -0700 Subject: [Python-Dev] python compiler In-Reply-To: <4BB9FDAC.10205@voidspace.org.uk> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: >> for a college project, I proposed to create a compiler for python. I've >> read something about it and maybe I saw that made a bad choice. I hear >> everyone's opinion respond. >> I don't think everyone thinks this is a bad idea -- for instance, those of us working on Cython [1], which is itself a descendant of Pyrex [2]. :) > Python itself is a highly dynamic language and not amenable to direct > compilation. Instead modern just-in-time compiler technology is seen as the > way to improve Python performance. Projects that are doing this are PyPy and > Unladen Swallow. A static subset of Python can be statically compiled, > projects that do that include RPython (part of PyPy) and ShedSkin. These are > not really Python though, just Python like languages that happen to be valid > subsets of Python. > It's true that JIT compilation really opens up a whole world of possibilities that Cython currently can't touch. On the other hand, for some kinds of Python code -- especially, for example, things related to scientific computing or mathematics -- Cython's a quick road to massive speedups, because a little bit of static typing can go a *long* way. It's true that Cython doesn't yet support the full Python syntax, but this is considered a bug -- we're working hard on being able to compile all of Python soon. -cc [1] http://www.cython.org/ [2] http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ From fijall at gmail.com Mon Apr 5 17:49:18 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Apr 2010 09:49:18 -0600 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: On Mon, Apr 5, 2010 at 9:31 AM, Craig Citro wrote: >>> for a college project, I proposed to create a compiler for python. I've >>> read something about it and maybe I saw that made a bad choice. I hear >>> everyone's opinion respond. >>> > > I don't think everyone thinks this is a bad idea -- for instance, > those of us working on Cython [1], which is itself a descendant of > Pyrex [2]. :) > I hate to remind you but Cython is *not* python. It does not even plan to support all of the parts which are considered python semantics (like tracebacks and frames). Cheers, fijal From craigcitro at gmail.com Mon Apr 5 19:32:11 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 5 Apr 2010 10:32:11 -0700 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: > I hate to remind you but Cython is *not* python. It does not even plan > to support all of the parts which are considered python semantics > (like tracebacks and frames). > It's true -- we basically compile to C + the Python/C API, depending on CPython being around for runtime support, and I don't see that changing anytime soon. (I don't think I tried to claim that we were a full Python implementation in my original email ...) I'm curious about the bit you mention, though -- is constructing a call frame for every Python call really part of the semantics, and not just a CPython implementation detail? (I've never played with Jython or IronPython to know if they do this.) We actually *do* construct all the call frames when doing profiling, so we could turn this on if we needed to for a "strict" mode, but usually the additional runtime speedup is more desirable. Independent of this, the OP was asking about working on something as part of a school-related project. I think that if you're looking to see how a Python to C compiler works, you could get quite a bit from checking out Cython and/or Pyrex, even if your real goal was to create a Python implementation independent of CPython. -cc From guido at python.org Mon Apr 5 20:03:30 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Apr 2010 11:03:30 -0700 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: On Mon, Apr 5, 2010 at 8:49 AM, Maciej Fijalkowski wrote: >> I hate to remind you but Cython is *not* python. It does not even plan >> to support all of the parts which are considered python semantics >> (like tracebacks and frames). On Mon, Apr 5, 2010 at 10:32 AM, Craig Citro wrote: > It's true -- we basically compile to C + the Python/C API, depending > on CPython being around for runtime support, and I don't see that > changing anytime soon. (I don't think I tried to claim that we were a > full Python implementation in my original email ...) There has been some contentious debate about this in the past, where a Cython developer(s?) insisted Cython be listed among the "Python implementations" somewhere, on a par with IronPython, Jython and PyPy. This does not seem the right place to list Cython to me. (Much though I admire Cython's accomplishments!) Insofar as the OP asked about "a compiler for Python" and you pointed to Cython as an example, your post could easily be misunderstood as an attempt to reopen that debate. > I'm curious about > the bit you mention, though -- is constructing a call frame for every > Python call really part of the semantics, and not just a CPython > implementation detail? (I've never played with Jython or IronPython to > know if they do this.) We actually *do* construct all the call frames > when doing profiling, so we could turn this on if we needed to for a > "strict" mode, but usually the additional runtime speedup is more > desirable. Beign able to access call frames via tracebacks is certainly part of the standard semantics, and a proper Python implementation should go through great lengths to get this interface right (as IronPython, Jython and PyPy indeed do). > Independent of this, the OP was asking about working on something as > part of a school-related project. I think that if you're looking to > see how a Python to C compiler works, you could get quite a bit from > checking out Cython and/or Pyrex, even if your real goal was to create > a Python implementation independent of CPython. -- --Guido van Rossum (python.org/~guido) From dreamingforward at gmail.com Mon Apr 5 21:23:39 2010 From: dreamingforward at gmail.com (average) Date: Mon, 5 Apr 2010 14:23:39 -0500 Subject: [Python-Dev] nonlocals() function? In-Reply-To: <201004050944.58416.steve@pearwood.info> References: <201004050944.58416.steve@pearwood.info> Message-ID: > globals() and locals() return dicts mapping names to objects. Damn, I totally pulled a *?!* on that one. I should have pulled out my Python reference. I was thinking of dir() and thought that these functions were similar. Apologies for that. However, I still do believe that as a general practice (not necessarily a Python one), return types should be the most specific type which will hold the data. > I'm not sure why you think that the globals() and locals() .... > You say > they should be properties and attributes, but properties of what? They would be properties of the implicit "main" [function] object. Sorry again for not checking before opening my "mouth". marcos From craigcitro at gmail.com Mon Apr 5 21:47:26 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 5 Apr 2010 12:47:26 -0700 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: > There has been some contentious debate about this in the past, where a > Cython developer(s?) insisted Cython be listed among the "Python > implementations" somewhere, on a par with IronPython, Jython and PyPy. > This does not seem the right place to list Cython to me. (Much though > I admire Cython's accomplishments!) > > Insofar as the OP asked about "a compiler for Python" and you pointed > to Cython as an example, your post could easily be misunderstood as an > attempt to reopen that debate. > Ahh. Well, I can't speak for the rest of the Cython folks, but let me say: I agree, Cython can't stand on its own without CPython, and there are a ton of other things I'm personally interested in doing long before we try to change that. Sorry if I was accidentally fanning any dying embers. :) > Beign able to access call frames via tracebacks is certainly part of > the standard semantics, and a proper Python implementation should go > through great lengths to get this interface right (as IronPython, > Jython and PyPy indeed do). > Is the requirement just the construction of full tracebacks in the event of an exception? Because Cython does that right now. In the event of an exception, the Python call frames are constructed as the C call stack is unwound. I thought Maciej was suggesting that having full Python frames for any Python call was part of the semantics, which is what we only do in the case of a profiling run. I'm really not trying to belabor the point -- but if there's something Cython should be doing and isn't, it's always good to hear about it. (Feel free to point me to an appropriate section of the docs ...) -cc From tjreedy at udel.edu Mon Apr 5 22:10:02 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 05 Apr 2010 16:10:02 -0400 Subject: [Python-Dev] python compiler In-Reply-To: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: On 4/5/2010 10:54 AM, willian at ufpa.br wrote: > for a college project, I proposed to create a compiler for python. I've > read something about it and maybe I saw that made a bad choice. I hear > everyone's opinion respond. If you want to do something useful, pick an existing project (several have already been named) that intrigues you and work on filling in one of the holes. I can imagine that this might be as educational as, for instance, compiling a toy subset of Python. tjr From fijall at gmail.com Mon Apr 5 22:11:33 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Apr 2010 14:11:33 -0600 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: On Mon, Apr 5, 2010 at 1:47 PM, Craig Citro wrote: >> There has been some contentious debate about this in the past, where a >> Cython developer(s?) insisted Cython be listed among the "Python >> implementations" somewhere, on a par with IronPython, Jython and PyPy. >> This does not seem the right place to list Cython to me. (Much though >> I admire Cython's accomplishments!) >> >> Insofar as the OP asked about "a compiler for Python" and you pointed >> to Cython as an example, your post could easily be misunderstood as an >> attempt to reopen that debate. >> > > Ahh. Well, I can't speak for the rest of the Cython folks, but let me > say: I agree, Cython can't stand on its own without CPython, and there > are a ton of other things I'm personally interested in doing long > before we try to change that. Sorry if I was accidentally fanning any > dying embers. :) > >> Beign able to access call frames via tracebacks is certainly part of >> the standard semantics, and a proper Python implementation should go >> through great lengths to get this interface right (as IronPython, >> Jython and PyPy indeed do). >> > > Is the requirement just the construction of full tracebacks in the > event of an exception? Because Cython does that right now. In the > event of an exception, the Python call frames are constructed as the C > call stack is unwound. I thought Maciej was suggesting that having > full Python frames for any Python call was part of the semantics, > which is what we only do in the case of a profiling run. I'm really > not trying to belabor the point -- but if there's something Cython > should be doing and isn't, it's always good to hear about it. (Feel > free to point me to an appropriate section of the docs ...) > However those frames are not completely like Python frames - they don't let you look into locals for example. It's a cool feature though that it exists at all, I did not know that. Cheers, fijal From fuzzyman at voidspace.org.uk Mon Apr 5 22:21:15 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 05 Apr 2010 21:21:15 +0100 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: <4BBA463B.7080804@voidspace.org.uk> On 05/04/2010 21:10, Terry Reedy wrote: > On 4/5/2010 10:54 AM, willian at ufpa.br wrote: >> for a college project, I proposed to create a compiler for python. I've >> read something about it and maybe I saw that made a bad choice. I hear >> everyone's opinion respond. > > If you want to do something useful, pick an existing project (several > have already been named) that intrigues you and work on filling in one > of the holes. I can imagine that this might be as educational as, for > instance, compiling a toy subset of Python. The type inferencing algorithm of ShedSkin might be particularly interesting to study and improve. They also need a Windows maintainer... Michael > > tjr > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fijall at gmail.com Mon Apr 5 22:28:21 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Apr 2010 14:28:21 -0600 Subject: [Python-Dev] python compiler In-Reply-To: <4BBA463B.7080804@voidspace.org.uk> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BBA463B.7080804@voidspace.org.uk> Message-ID: On Mon, Apr 5, 2010 at 2:21 PM, Michael Foord wrote: > On 05/04/2010 21:10, Terry Reedy wrote: >> >> On 4/5/2010 10:54 AM, willian at ufpa.br wrote: >>> >>> for a college project, I proposed to create a compiler for python. I've >>> read something about it and maybe I saw that made a bad choice. I hear >>> everyone's opinion respond. >> >> If you want to do something useful, pick an existing project (several have >> already been named) that intrigues you and work on filling in one of the >> holes. I can imagine that this might be as educational as, for instance, >> compiling a toy subset of Python. > > The type inferencing algorithm of ShedSkin might be particularly interesting > to study and improve. They also need a Windows maintainer... > > Michael > Not sure if you went to college Michael, but "Windows maintainer" is not something you can bribe your professors with ;-) Cheers, fijal From martin at v.loewis.de Mon Apr 5 23:03:59 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Apr 2010 23:03:59 +0200 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: <4BBA503F.4090006@v.loewis.de> > Is the requirement just the construction of full tracebacks in the > event of an exception? Because Cython does that right now. In the > event of an exception, the Python call frames are constructed as the C > call stack is unwound. I thought Maciej was suggesting that having > full Python frames for any Python call was part of the semantics, > which is what we only do in the case of a profiling run. I'm really > not trying to belabor the point -- but if there's something Cython > should be doing and isn't, it's always good to hear about it. (Feel > free to point me to an appropriate section of the docs ...) I guess being able to do inspect.currentframe() is a "SHOULD" requirement for Python implementations. It's clear that it will be difficult to provide, so portable applications SHOULD NOT rely on it working correctly. Regards, Martin From fijall at gmail.com Mon Apr 5 23:07:49 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Apr 2010 15:07:49 -0600 Subject: [Python-Dev] python compiler In-Reply-To: <4BBA503F.4090006@v.loewis.de> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> <4BBA503F.4090006@v.loewis.de> Message-ID: On Mon, Apr 5, 2010 at 3:03 PM, "Martin v. L?wis" wrote: >> Is the requirement just the construction of full tracebacks in the >> event of an exception? Because Cython does that right now. In the >> event of an exception, the Python call frames are constructed as the C >> call stack is unwound. I thought Maciej was suggesting that having >> full Python frames for any Python call was part of the semantics, >> which is what we only do in the case of a profiling run. I'm really >> not trying to belabor the point -- but if there's something Cython >> should be doing and isn't, it's always good to hear about it. (Feel >> free to point me to an appropriate section of the docs ...) > > I guess being able to do inspect.currentframe() is a "SHOULD" > requirement for Python implementations. It's clear that it will be > difficult to provide, so portable applications SHOULD NOT rely on it > working correctly. > > Regards, > Martin > According to docs sys._getframe() is a impl-specific thing, however sys.exc_info() presenting correct interface of it's 3rd result (a traceback) is not. That's at least how I read those docs. Cheers, fijal From greg.ewing at canterbury.ac.nz Tue Apr 6 01:37:11 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 11:37:11 +1200 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: Message-ID: <4BBA7427.8030601@canterbury.ac.nz> Antoine Pitrou wrote: > Steve Bonner gmail.com> writes: > >>What do we think of adding a built-in nonlocals() function that would >>be similar to globals() and locals()? > > These scopes don't have parallel capabilities: Maybe it would be better to deprecate globals() and locals() and replace them with another function called something like scope(). It would return a mapping object that looks up names in the current scope. It could also improve on locals() by being writable. -- Greg From fuzzyman at voidspace.org.uk Tue Apr 6 01:29:06 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 06 Apr 2010 00:29:06 +0100 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: <4BBA7427.8030601@canterbury.ac.nz> References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: <4BBA7242.80507@voidspace.org.uk> On 06/04/2010 00:37, Greg Ewing wrote: > Antoine Pitrou wrote: >> Steve Bonner gmail.com> writes: >> >>> What do we think of adding a built-in nonlocals() function that would >>> be similar to globals() and locals()? >> >> These scopes don't have parallel capabilities: > > Maybe it would be better to deprecate globals() and locals() > and replace them with another function called something like > scope(). It would return a mapping object that looks up > names in the current scope. It could also improve on locals() > by being writable. > Well, not a bad idea but it belongs on python-ideas and would be covered by the language moratorium. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From greg.ewing at canterbury.ac.nz Tue Apr 6 01:44:51 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 11:44:51 +1200 Subject: [Python-Dev] python compiler In-Reply-To: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: <4BBA75F3.2030502@canterbury.ac.nz> willian at ufpa.br wrote: > for a college project, I proposed to create a compiler for python. I've > read something about it and maybe I saw that made a bad choice. I hear > everyone's opinion respond. I don't want to discourage you if you really want to try, but you need to be aware that you'd be taking on a very challenging project -- at least if you intend the compiled code to be substantially faster than interpretation. You should also check out what others have been doing in this area: PyPy, Cython, Unladen Swallow. -- Greg From solipsis at pitrou.net Tue Apr 6 01:35:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Apr 2010 23:35:24 +0000 (UTC) Subject: [Python-Dev] Scope object (Re: nonlocals() function?) References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > Maybe it would be better to deprecate globals() and locals() > and replace them with another function called something like > scope(). It is useful to distinguish between globals (i.e., module-level variables) and locals, so replacing them with scope() would not be better IMO. > It would return a mapping object that looks up > names in the current scope. It could also improve on locals() > by being writable. If you can prove that making locals() (or its replacement) writable doesn't complicate the interpreter core too much, then why not. Otherwise -1 :-) Regards Antoine. From greg.ewing at canterbury.ac.nz Tue Apr 6 02:03:27 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 12:03:27 +1200 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BB9FDAC.10205@voidspace.org.uk> Message-ID: <4BBA7A4F.7090608@canterbury.ac.nz> Craig Citro wrote: > In the > event of an exception, the Python call frames are constructed as the C > call stack is unwound. Although in Pyrex the frames have just enough info in them to find out the file name and line number -- the rest (f_stack, f_locals, etc.) are filled with dummy values. -- Greg From rnk at mit.edu Tue Apr 6 04:06:10 2010 From: rnk at mit.edu (Reid Kleckner) Date: Mon, 5 Apr 2010 22:06:10 -0400 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: On Mon, Apr 5, 2010 at 7:35 PM, Antoine Pitrou wrote: > If you can prove that making locals() (or its replacement) writable doesn't > complicate the interpreter core too much, then why not. Otherwise -1 :-) I think writable locals would significantly complicate the job of people trying to optimize Python. The current situation is that so long as a code object is compiled with certain flags and avoids using exec, then it is impossible to indirectly modify locals in a call frame without resorting to compiled code that mucks with the frame directly. It was very easy for us to check for these conditions, and if they were met, emit faster code. Collin implemented/optimized local variable access for unladen, so he would know more than I. If I remember correctly, the exec statement is going away in py3k, and calling exec() with one argument can modify the local scope. Therefore we'll probably have to do something more sophisticated anyway. :( This would impact PyPy, Jython, and the other implementations, so I would think twice about it. Reid From fijall at gmail.com Tue Apr 6 06:01:29 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Apr 2010 22:01:29 -0600 Subject: [Python-Dev] python compiler In-Reply-To: <4BBA75F3.2030502@canterbury.ac.nz> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BBA75F3.2030502@canterbury.ac.nz> Message-ID: On Mon, Apr 5, 2010 at 5:44 PM, Greg Ewing wrote: > willian at ufpa.br wrote: >> >> for a college project, I proposed to create a compiler for python. I've >> read something about it and maybe I saw that made a bad choice. I hear >> everyone's opinion respond. > > I don't want to discourage you if you really want to try, > but you need to be aware that you'd be taking on a very > challenging project -- at least if you intend the compiled > code to be substantially faster than interpretation. > > You should also check out what others have been doing in > this area: PyPy, Cython, Unladen Swallow. > Except none of the things mentioned above is actually a "Python compiler". There are however good reasons why not and maybe that's worth checking. Cheers, fijal From cesare.di.mauro at gmail.com Tue Apr 6 08:25:08 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Tue, 6 Apr 2010 09:25:08 +0300 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: 2010/4/6 Antoine Pitrou > Greg Ewing canterbury.ac.nz> writes: > > > > Maybe it would be better to deprecate globals() and locals() > > and replace them with another function called something like > > scope(). > > It is useful to distinguish between globals (i.e., module-level variables) > and > locals, so replacing them with scope() would not be better IMO. > > > It would return a mapping object that looks up > > names in the current scope. It could also improve on locals() > > by being writable. > > If you can prove that making locals() (or its replacement) writable doesn't > complicate the interpreter core too much, then why not. Otherwise -1 :-) > > Regards > > Antoine. > It will certainly. There's MUCH that can be optimized to let CPython squeeze more performance from static analysis (even a gross one) on locals. Example: def f(): a = 1 b = 2 return a + b can be reduced to something similar to: def f(): a = 1 b = 2 return 3 and, more aggressively, like: def f(): return 3 They are just "dummy" examples, but can make it clear how far optimizations can go with static analysis on locals. Python is a language that make it possible to use such analysis at compile time, and I think it is a very good thing. Obviously the last example brings questions regards the language semantic: is it right to suppress "unused" or "not useful" local variables? A "conservative" answer will be clearly NO. But I hope that a future language specification will fix some aspects, putting clear what you can expect from the language itself, and what is closet to the implementation. Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Apr 6 11:13:08 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 21:13:08 +1200 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: <4BBAFB24.7030805@canterbury.ac.nz> Reid Kleckner wrote: > If I remember correctly, the exec statement is going away in py3k, and > calling exec() with one argument can modify the local scope. I've been kind of wondering what the deal is with exec in py3. I always thought the reason for making exec a statement was so that locals optimisation could be turned off in its presence, so I'm not sure how py3 is getting away with making it a function. Anyhow, it seems to me that as long as locals() or whatever might replace it is able to find the existing value of a local, it should also be able to change that value, wherever it happens to be stored. I suppose that might fail if an optimiser decides to keep multiple copies of a local for some reason, though. But even if it has to be read-only, I still think a view object would be a more py3ish way of handling locals() and the like. You might only want access to a few locals, in which case building a dict containing all of them would be wasteful. -- Greg -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 6 11:22:31 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 21:22:31 +1200 Subject: [Python-Dev] python compiler In-Reply-To: References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> <4BBA75F3.2030502@canterbury.ac.nz> Message-ID: <4BBAFD57.9030802@canterbury.ac.nz> Maciej Fijalkowski wrote: > Except none of the things mentioned above is actually a "Python > compiler". No, but they grapple with many of the same issues that a Python compiler would face, and it would be informative to see how they tackle those issues. If you want to advance the state of the art, you first need to find out what the state of the art is. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 6 11:28:25 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Apr 2010 21:28:25 +1200 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: <4BBA7427.8030601@canterbury.ac.nz> Message-ID: <4BBAFEB9.8000002@canterbury.ac.nz> Cesare Di Mauro wrote: > It will certainly. There's MUCH that can be optimized to let CPython > squeeze more performance from static analysis (even a gross one) on locals. But can the existing locals() function be implemented in the face of such optimisations? If it can, then a "locals view" object shouldn't be too much harder. If it can't, then you have already given up full CPython compatibility. -- Greg From techtonik at gmail.com Tue Apr 6 11:40:35 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 6 Apr 2010 12:40:35 +0300 Subject: [Python-Dev] Code coverage metrics Message-ID: Where can I find public reports with Python tests code coverage? -- anatoly t. From cournape at gmail.com Tue Apr 6 11:41:56 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Apr 2010 18:41:56 +0900 Subject: [Python-Dev] python compiler In-Reply-To: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> References: <964662d5539265fef373cb8e7b0ccb5c@ufpa.br> Message-ID: On Mon, Apr 5, 2010 at 11:54 PM, wrote: > for a college project, I proposed to create a compiler for python. I've > read something about it and maybe I saw that made a bad choice. I hear > everyone's opinion respond. Depending on your taste, you may want to tackle something like a static analyser for python. This is not a compiler proper, but it could potentially be more useful than yet another compiler compiling 50 % of python, and you would get some results more quickly (no need to generate code, etc...). See e.g. http://bugs.jython.org/issue1541 for an actual implementation on a similar idea (but for jython), cheers, David From orsenthil at gmail.com Tue Apr 6 11:50:51 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Tue, 6 Apr 2010 15:20:51 +0530 Subject: [Python-Dev] Code coverage metrics In-Reply-To: References: Message-ID: <20100406095051.GA5946@remy> On Tue, Apr 06, 2010 at 12:40:35PM +0300, anatoly techtonik wrote: > Where can I find public reports with Python tests code coverage? Here: http://coverage.livinglogic.de/ -- Senthil From cesare.di.mauro at gmail.com Tue Apr 6 12:03:54 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Tue, 6 Apr 2010 13:03:54 +0300 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: <4BBAFEB9.8000002@canterbury.ac.nz> References: <4BBA7427.8030601@canterbury.ac.nz> <4BBAFEB9.8000002@canterbury.ac.nz> Message-ID: 2010/4/6 Greg Ewing > Cesare Di Mauro wrote: > > It will certainly. There's MUCH that can be optimized to let CPython >> squeeze more performance from static analysis (even a gross one) on locals. >> > > But can the existing locals() function be implemented in > the face of such optimisations? > > If it can, then a "locals view" object shouldn't be too much > harder. > > If it can't, then you have already given up full CPython > compatibility. > > -- > Greg A read-only locals view can be a good comprise, because at least the first example I showed can be approached well. For the second example, there's no full compatibility with the current CPython implementation. But implementations can change over the time: we can clearly define that on future CPython versions no assumptions must be made about locals "usage", and in general about instructions generation. The most important thing is that the function f() does what is called to do: return the numeric constant 3. This gives us the opportunity to schedule more efficient optimizations, without losing generality about the language (only some weird tricks will not be supported). Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcea at jcea.es Tue Apr 6 12:21:35 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 06 Apr 2010 12:21:35 +0200 Subject: [Python-Dev] "-Wd" switch Message-ID: <4BBB0B2F.2050500@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Recently we added "-Wd" flags to buildbots. I was wondering about the effect of it. documentation doesn't help. I could study the code, but I guess other people can have the very same question and I think the answer should be in the archives, somewhere. Thanks. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS7sLL5lgi5GaxT1NAQL7pQP/ZbrBSm6Ojso3edRWZ+gz1IU+M+aN2dfU NCVAO4ZaXCVkZiS3YcUCy0sg8HahmsC50XpDAZC0r59+phOJazYaol6wz7ECxPe/ Rymu09nCi2Yu7GlWqB5SuLKWhxUVVqJvqfy2oJru8HfhTXj0dYa/hsUxdLUcZIB3 vNEr2VtBNhw= =VG9K -----END PGP SIGNATURE----- From larry at hastings.org Tue Apr 6 13:19:55 2010 From: larry at hastings.org (Larry Hastings) Date: Tue, 06 Apr 2010 04:19:55 -0700 Subject: [Python-Dev] "-Wd" switch In-Reply-To: <4BBB0B2F.2050500@jcea.es> References: <4BBB0B2F.2050500@jcea.es> Message-ID: <4BBB18DB.9030302@hastings.org> Jesus Cea wrote: > Recently we added "-Wd" flags to buildbots. I was wondering about the > effect of it. documentation doesn't help. > > I could study the code, but I guess other people can have the very same > question and I think the answer should be in the archives, somewhere. I studied the code ;) -Wd enables all warnings. It adds 'd' to sys.warnoptions, which in turn adds a new first entry to _warnings.filters which matches all warnings and enables the "default" behavior, which is to show it once per execution of the Python interpreter. For example, if you run "python -Wd" on the current trunk (2.7) and execute the statement "import bsddb" you get a PendingDeprecationWarning exception. Without -Wd that warning would be suppressed. Hope I didn't miss any important subtleties, //larry// From willian at ufpa.br Tue Apr 6 13:44:15 2010 From: willian at ufpa.br (willian at ufpa.br) Date: Tue, 6 Apr 2010 11:44:15 +0000 (UTC) Subject: [Python-Dev] Python and compilers Message-ID: <2846531d5737614a4248198f77fe55d9@ufpa.br> First, thank you for all opnion. Each one was considered. I think the better question would be: I have to develop a project that involves compilers, and being a fan of Python, I thought about making a compiler for it (most basic idea involving Pythin and compilers). But I saw that I can use what I learned from compilers not only to create a compiler. What is the area of developing the Python interpreter that I could build my project, and please give me interesting ideas for the project. From techtonik at gmail.com Tue Apr 6 13:50:28 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 6 Apr 2010 14:50:28 +0300 Subject: [Python-Dev] Code coverage metrics In-Reply-To: <20100406095051.GA5946@remy> References: <20100406095051.GA5946@remy> Message-ID: On Tue, Apr 6, 2010 at 12:50 PM, Senthil Kumaran wrote: >> Where can I find public reports with Python tests code coverage? > > Here: > > http://coverage.livinglogic.de/ Thank you. What is the status of getting these stats on python.org? -- anatoly t. From thobes at gmail.com Tue Apr 6 13:50:48 2010 From: thobes at gmail.com (Tobias Ivarsson) Date: Tue, 6 Apr 2010 13:50:48 +0200 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: <4BBAFB24.7030805@canterbury.ac.nz> References: <4BBA7427.8030601@canterbury.ac.nz> <4BBAFB24.7030805@canterbury.ac.nz> Message-ID: On Tue, Apr 6, 2010 at 11:13 AM, Greg Ewing wrote: > Reid Kleckner wrote: > > If I remember correctly, the exec statement is going away in py3k, and >> calling exec() with one argument can modify the local scope. >> > > I've been kind of wondering what the deal is with exec in py3. > I always thought the reason for making exec a statement was so > that locals optimisation could be turned off in its presence, > so I'm not sure how py3 is getting away with making it a > function. > It looks like py3 does not allow exec to modify the locals: $ python3 Python 3.1.1 (r311:74543, Aug 24 2009, 18:44:04) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> def x(a): ... exec(a) ... return a ... >>> x("a = 5") 'a = 5' >>> # the above statement would have returned 5 if the locals had been modified > > Anyhow, it seems to me that as long as locals() or whatever > might replace it is able to find the existing value of a local, > it should also be able to change that value, wherever it > happens to be stored. > > I suppose that might fail if an optimiser decides to keep > multiple copies of a local for some reason, though. > > But even if it has to be read-only, I still think a view object > would be a more py3ish way of handling locals() and the like. > You might only want access to a few locals, in which case > building a dict containing all of them would be wasteful. > > -- > Greg > > > -- > Greg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thobes%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Tue Apr 6 14:02:47 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 06 Apr 2010 13:02:47 +0100 Subject: [Python-Dev] Python and compilers In-Reply-To: <2846531d5737614a4248198f77fe55d9@ufpa.br> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> Message-ID: <4BBB22E7.6060209@voidspace.org.uk> On 06/04/2010 12:44, willian at ufpa.br wrote: > First, thank you for all opnion. Each one was considered. > I think the better question would be: > I have to develop a project that involves compilers, and being a fan of > Python, I thought about making a compiler for it (most basic idea involving > Pythin and compilers). But I saw that I can use what I learned from > compilers not only to create a compiler. What is the area of developing the > Python interpreter that I could build my project, and please give me > interesting ideas for the project. > Well, you are now thoroughly off topic for the python-dev mailing list - so I suggest you ask your question on comp.lang.python or some other Python list. *However*, a project that would be interesting - and that I have wanted to do in order to program microcontrollers with *very* small memory address spaces [1] - would be to compile a static subset of Python down to C. You would need to do type inferencing and support only a basic minimum of the built-in types (and in fact I would start with perhaps functions only), but it could be fun. It would not be Python however, merely Python inspired. Alternatively a general type-inferencing algorithm for Python, such as the one used by ShedSkin, could be interesting. All the best, Michael [1] Smaller devices than those targetted by http://code.google.com/p/python-on-a-chip/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From solipsis at pitrou.net Tue Apr 6 15:45:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Apr 2010 13:45:44 +0000 (UTC) Subject: [Python-Dev] Code coverage metrics References: <20100406095051.GA5946@remy> Message-ID: Senthil Kumaran gmail.com> writes: > > On Tue, Apr 06, 2010 at 12:40:35PM +0300, anatoly techtonik wrote: > > Where can I find public reports with Python tests code coverage? > > Here: > > http://coverage.livinglogic.de/ The fact that the log shows some test failures isn't very comforting. Regards Antoine. From g.brandl at gmx.net Tue Apr 6 18:31:58 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Apr 2010 18:31:58 +0200 Subject: [Python-Dev] Code coverage metrics In-Reply-To: References: <20100406095051.GA5946@remy> Message-ID: Am 06.04.2010 13:50, schrieb anatoly techtonik: > On Tue, Apr 6, 2010 at 12:50 PM, Senthil Kumaran wrote: >>> Where can I find public reports with Python tests code coverage? >> >> Here: >> >> http://coverage.livinglogic.de/ > > Thank you. What is the status of getting these stats on python.org? Wouldn't "status" imply that there is a plan to do so? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From brett at python.org Tue Apr 6 20:28:12 2010 From: brett at python.org (Brett Cannon) Date: Tue, 6 Apr 2010 11:28:12 -0700 Subject: [Python-Dev] "-Wd" switch In-Reply-To: <4BBB18DB.9030302@hastings.org> References: <4BBB0B2F.2050500@jcea.es> <4BBB18DB.9030302@hastings.org> Message-ID: On Tue, Apr 6, 2010 at 04:19, Larry Hastings wrote: > Jesus Cea wrote: > >> Recently we added "-Wd" flags to buildbots. I was wondering about the >> effect of it. documentation doesn't help. >> >> I could study the code, but I guess other people can have the very same >> question and I think the answer should be in the archives, somewhere. >> > > I studied the code ;) > > -Wd enables all warnings. It adds 'd' to sys.warnoptions, which in turn > adds a new first entry to _warnings.filters which matches all warnings and > enables the "default" behavior, which is to show it once per execution of > the Python interpreter. > > For example, if you run "python -Wd" on the current trunk (2.7) and execute > the statement "import bsddb" you get a PendingDeprecationWarning exception. > Without -Wd that warning would be suppressed. > > Hope I didn't miss any important subtleties, > > Nope, you got it right. A little bit of documentation is in 2.7: http://docs.python.org/dev/library/warnings.html#updating-code-for-new-versions-of-python . > > //larry// > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Apr 6 20:41:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Apr 2010 18:41:04 +0000 (UTC) Subject: [Python-Dev] "-Wd" switch References: <4BBB0B2F.2050500@jcea.es> <4BBB18DB.9030302@hastings.org> Message-ID: Brett Cannon python.org> writes: > > > Nope, you got it right. A little bit of documentation is in > 2.7:?http://docs.python.org/dev/library/warnings.html#updating-code-for- It is a bit disturbing, though, that "-Wdefault" isn't the default setting. How could that oddity be solved? Regards Antoine. From techtonik at gmail.com Tue Apr 6 21:36:14 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 6 Apr 2010 22:36:14 +0300 Subject: [Python-Dev] Code coverage metrics In-Reply-To: References: <20100406095051.GA5946@remy> Message-ID: On Tue, Apr 6, 2010 at 7:31 PM, Georg Brandl wrote: > >>>> Where can I find public reports with Python tests code coverage? >>> >>> Here: >>> >>> http://coverage.livinglogic.de/ >> >> Thank you. What is the status of getting these stats on python.org? > > Wouldn't "status" imply that there is a plan to do so? It is not that hard to create that plan given that there is a fair amount of developers who care about code coverage. -- anatoly t. From ctb at msu.edu Tue Apr 6 21:42:12 2010 From: ctb at msu.edu (C. Titus Brown) Date: Tue, 6 Apr 2010 12:42:12 -0700 Subject: [Python-Dev] Code coverage metrics In-Reply-To: References: <20100406095051.GA5946@remy> Message-ID: <20100406194212.GB11799@idyll.org> On Tue, Apr 06, 2010 at 10:36:14PM +0300, anatoly techtonik wrote: > On Tue, Apr 6, 2010 at 7:31 PM, Georg Brandl wrote: > > > >>>> Where can I find public reports with Python tests code coverage? > >>> > >>> Here: > >>> > >>> http://coverage.livinglogic.de/ > >> > >> Thank you. What is the status of getting these stats on python.org? > > > > Wouldn't "status" imply that there is a plan to do so? > > It is not that hard to create that plan given that there is a fair > amount of developers who care about code coverage. Anatoly, nonetheless, I don't know of any plan :). I keep on shaving yaks when trying to get test coverage posted somewhere; it's actually not a trivial problem to set it up automatically and reliably, in my experience. If you are interested in volunteering to help, I'm sure it would be appreciated -- I suspect the right place to start would be get test coverage running reproducibly and reliably on your own machines and posted somewhere; then you could ask for permissions to post it somewhere on *.python.org. (I don't see why we care where it's posted, but that's what you asked about.) I keep on running into technical barriers in getting cross-platform code coverage analysis working, which would be quite valuable; it's easy to get it working once, but to keep it working is a maintenance task that involves regular effort, again, in my experience. best, --titus -- C. Titus Brown, ctb at msu.edu From brett at python.org Tue Apr 6 23:13:07 2010 From: brett at python.org (Brett Cannon) Date: Tue, 6 Apr 2010 14:13:07 -0700 Subject: [Python-Dev] Code coverage metrics In-Reply-To: <20100406194212.GB11799@idyll.org> References: <20100406095051.GA5946@remy> <20100406194212.GB11799@idyll.org> Message-ID: On Tue, Apr 6, 2010 at 12:42, C. Titus Brown wrote: > On Tue, Apr 06, 2010 at 10:36:14PM +0300, anatoly techtonik wrote: > > On Tue, Apr 6, 2010 at 7:31 PM, Georg Brandl wrote: > > > > > >>>> Where can I find public reports with Python tests code coverage? > > >>> > > >>> Here: > > >>> > > >>> http://coverage.livinglogic.de/ > > >> > > >> Thank you. What is the status of getting these stats on python.org? > > > > > > Wouldn't "status" imply that there is a plan to do so? > > > > It is not that hard to create that plan given that there is a fair > > amount of developers who care about code coverage. > > Anatoly, > > nonetheless, I don't know of any plan :). > > I keep on shaving yaks when trying to get test coverage posted somewhere; > it's actually not a trivial problem to set it up automatically and > reliably, in my experience. If you are interested in volunteering to > help, I'm sure it would be appreciated -- I suspect the right place to > start would be get test coverage running reproducibly and reliably on > your own machines and posted somewhere; then you could ask for permissions > to post it somewhere on *.python.org. (I don't see why we care where it's > posted, but that's what you asked about.) > > I keep on running into technical barriers in getting cross-platform code > coverage analysis working, which would be quite valuable; it's easy to > get it working once, but to keep it working is a maintenance task that > involves regular effort, again, in my experience. > And just in general, it requires someone stepping forward to do it. While there might be several developers who find test coverage important (me included), it's a matter of taking the time to set up something that will work. For me personally that is more time than I have for my volunteer work for Python as I have other things that I put at a higher priority. It's an overused, worn out statement, but it's open source so stuff only gets done when someone REALLY has the desire and time to do something. I'm grateful that Walter has done what he has. If someone steps forward to write up instructions on how to run figleaf or coverage.py over the test suite then that would be appreciated. But just because something isn't technically hard, doesn't mean it isn't hard from the perspective of time and motivation. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Apr 6 23:17:28 2010 From: brett at python.org (Brett Cannon) Date: Tue, 6 Apr 2010 14:17:28 -0700 Subject: [Python-Dev] "-Wd" switch In-Reply-To: References: <4BBB0B2F.2050500@jcea.es> <4BBB18DB.9030302@hastings.org> Message-ID: On Tue, Apr 6, 2010 at 11:41, Antoine Pitrou wrote: > Brett Cannon python.org> writes: > > > > > > Nope, you got it right. A little bit of documentation is in > > 2.7: http://docs.python.org/dev/library/warnings.html#updating-code-for- > > It is a bit disturbing, though, that "-Wdefault" isn't the default setting. > How could that oddity be solved? > Well, it needs to be understood that `-Wdefault` is shorthand for `-W default:.*:Warning:.*` plus whatever the wildcard is for line number. So the "default" doesn't mean "default for when the interpreter starts up" but "the default action for any warning". It's a shorthand that unfortunately reads on an odd fashion when you do not realize what the expanded value means. Best you can do is in the docs explain what it's shorthand for. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Apr 6 23:51:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Apr 2010 09:51:16 +1200 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBB22E7.6060209@voidspace.org.uk> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBB22E7.6060209@voidspace.org.uk> Message-ID: <4BBBACD4.4000906@canterbury.ac.nz> Michael Foord wrote: > *However*, a project that would be interesting - and that I have wanted > to do in order to program microcontrollers with *very* small memory > address spaces [1] - would be to compile a static subset of Python down > to C. That would be an excellent project -- if the result were successful, I'd be interested in using it! -- Greg From walter at livinglogic.de Tue Apr 6 23:45:34 2010 From: walter at livinglogic.de (Walter Doerwald) Date: Tue, 06 Apr 2010 23:45:34 +0200 Subject: [Python-Dev] Code coverage metrics In-Reply-To: <20100406095051.GA5946@remy> References: <20100406095051.GA5946@remy> Message-ID: <4BBBAB7E.2070508@livinglogic.de> On 06.04.2010 11:50, Senthil Kumaran wrote: > On Tue, Apr 06, 2010 at 12:40:35PM +0300, anatoly techtonik wrote: >> Where can I find public reports with Python tests code coverage? > > Here: > > http://coverage.livinglogic.de/ And the script that generates that output is available from the cheeseshop: http://pypi.python.org/pypi/pycoco Servus, Walter From martin at v.loewis.de Wed Apr 7 00:31:06 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Apr 2010 00:31:06 +0200 Subject: [Python-Dev] Python and compilers In-Reply-To: <2846531d5737614a4248198f77fe55d9@ufpa.br> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> Message-ID: <4BBBB62A.7000404@v.loewis.de> willian at ufpa.br wrote: > First, thank you for all opnion. Each one was considered. > I think the better question would be: > I have to develop a project that involves compilers, and being a fan of > Python, I thought about making a compiler for it (most basic idea involving > Pythin and compilers). But I saw that I can use what I learned from > compilers not only to create a compiler. What is the area of developing the > Python interpreter that I could build my project, and please give me > interesting ideas for the project. I don't think the question is necessarily off-topic. I can propose two projects, related to Python core: - 2to3 pattern compiler: 2to3 currently uses an interpreter for pattern matching. It does "compile" the patterns into some intermediate form, however, that is actually interpreted with an interpreter written in Python (actually, it is self-interpreted). It might be interesting to compile the pattern into actual Python code, with the hope of it executing faster than it does now (and yes, I proposed a similar, but different GSoC topic also; the two approaches are orthogonal, though). - IDLE code completion. Currently, IDLE has some form of code completion, which is fairly limited. It might be useful to produce a better code completion library, one that works more statically and less based on introspection. In particular, optimistic type inference might help (optimistic in the sense "if foo has a method .isalpha, it probably is a string). In code completion, exact type inference isn't necessary; giving a superset (i.e. a union type) might still be helpful. In addition, to-python compilers may also be interesting in various HTML templating languages, e.g. Django templating, Zope page templates, and the like (although some of them already have compilers of some form on their own). HTH, Martin From fuzzyman at voidspace.org.uk Wed Apr 7 00:34:50 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 06 Apr 2010 23:34:50 +0100 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBBB62A.7000404@v.loewis.de> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBBB62A.7000404@v.loewis.de> Message-ID: <4BBBB70A.5060604@voidspace.org.uk> On 06/04/2010 23:31, "Martin v. L?wis" wrote: > willian at ufpa.br wrote: > >> First, thank you for all opnion. Each one was considered. >> I think the better question would be: >> I have to develop a project that involves compilers, and being a fan of >> Python, I thought about making a compiler for it (most basic idea involving >> Pythin and compilers). But I saw that I can use what I learned from >> compilers not only to create a compiler. What is the area of developing the >> Python interpreter that I could build my project, and please give me >> interesting ideas for the project. >> > I don't think the question is necessarily off-topic. > > I can propose two projects, related to Python core: > > - 2to3 pattern compiler: 2to3 currently uses an interpreter for pattern > matching. It does "compile" the patterns into some intermediate form, > however, that is actually interpreted with an interpreter written in > Python (actually, it is self-interpreted). It might be interesting to > compile the pattern into actual Python code, with the hope of it > executing faster than it does now (and yes, I proposed a similar, but > different GSoC topic also; the two approaches are orthogonal, though). > > - IDLE code completion. Currently, IDLE has some form of code > completion, which is fairly limited. It might be useful to produce a > better code completion library, one that works more statically and > less based on introspection. In particular, optimistic type inference > might help (optimistic in the sense "if foo has a method .isalpha, it > probably is a string). In code completion, exact type inference isn't > necessary; giving a superset (i.e. a union type) might still be > helpful. > > This would be very useful to many Python IDE projects. Getting it right is one thing, getting it fast enough to be useful is another (i.e. it is a difficult problem) - but yes, could be both interesting and useful. Michael > In addition, to-python compilers may also be interesting in various HTML > templating languages, e.g. Django templating, Zope page templates, and > the like (although some of them already have compilers of some form on > their own). > > HTH, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Wed Apr 7 00:45:57 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 06 Apr 2010 23:45:57 +0100 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBBB70A.5060604@voidspace.org.uk> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBBB62A.7000404@v.loewis.de> <4BBBB70A.5060604@voidspace.org.uk> Message-ID: <4BBBB9A5.3040702@voidspace.org.uk> On 06/04/2010 23:34, Michael Foord wrote: > On 06/04/2010 23:31, "Martin v. L?wis" wrote: >> willian at ufpa.br wrote: >>> First, thank you for all opnion. Each one was considered. >>> I think the better question would be: >>> I have to develop a project that involves compilers, and being a fan of >>> Python, I thought about making a compiler for it (most basic idea >>> involving >>> Pythin and compilers). But I saw that I can use what I learned from >>> compilers not only to create a compiler. What is the area of >>> developing the >>> Python interpreter that I could build my project, and please give me >>> interesting ideas for the project. >> I don't think the question is necessarily off-topic. >> >> I can propose two projects, related to Python core: >> >> - 2to3 pattern compiler: 2to3 currently uses an interpreter for pattern >> matching. It does "compile" the patterns into some intermediate form, >> however, that is actually interpreted with an interpreter written in >> Python (actually, it is self-interpreted). It might be interesting to >> compile the pattern into actual Python code, with the hope of it >> executing faster than it does now (and yes, I proposed a similar, but >> different GSoC topic also; the two approaches are orthogonal, >> though). >> >> - IDLE code completion. Currently, IDLE has some form of code >> completion, which is fairly limited. It might be useful to produce a >> better code completion library, one that works more statically and >> less based on introspection. In particular, optimistic type inference >> might help (optimistic in the sense "if foo has a method .isalpha, it >> probably is a string). In code completion, exact type inference isn't >> necessary; giving a superset (i.e. a union type) might still be >> helpful. >> > > This would be very useful to many Python IDE projects. Getting it > right is one thing, getting it fast enough to be useful is another > (i.e. it is a difficult problem) - but yes, could be both interesting > and useful. A good basis for a project like would be PySmell (by Orestis Markou), which intended to provide this but was never completed: http://code.google.com/p/pysmell/ It would make a great gsoc project as well. Michael > > Michael > > >> In addition, to-python compilers may also be interesting in various HTML >> templating languages, e.g. Django templating, Zope page templates, and >> the like (although some of them already have compilers of some form on >> their own). >> >> HTH, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From benjamin at python.org Wed Apr 7 01:09:06 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 6 Apr 2010 18:09:06 -0500 Subject: [Python-Dev] stabilizing for a release Message-ID: I propose that we freeze the trunk except for fixes for the buildbots. Thoughts? -- Regards, Benjamin From rdmurray at bitdance.com Wed Apr 7 01:15:24 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 06 Apr 2010 19:15:24 -0400 Subject: [Python-Dev] iso-2022 and issue 7472: question for the experts Message-ID: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> A long time ago (in a galaxy far far...no, wrong show) Er, as I was saying, a long time ago Barry applied a patch to email that went more or less like this: ndex: email/Encoders.py =================================================================== --- email/Encoders.py (revision 35918) +++ email/Encoders.py (revision 35919) @@ -84,7 +83,13 @@ try: orig.encode('ascii') except UnicodeError: - msg['Content-Transfer-Encoding'] = '8bit' + # iso-2022-* is non-ASCII but still 7-bit + charset = msg.get_charset() + output_cset = charset and charset.output_charset + if output_cset and output_cset.lower().startswith('iso-2202-'): + msg['Content-Transfer-Encoding'] = '7bit' + else: + msg['Content-Transfer-Encoding'] = '8bit' else: msg['Content-Transfer-Encoding'] = '7bit' Nobody noticed the typo (2202 instead of 2022) until Yukihiro reported it in the issue. No one has complained about the code not working. There are no unit tests that cover the code (all tests pass with or without the patch, and alternatively with or without correcting the typo). Reading the standards, it looks to me like either the ISO-2022 input will be 7-bit, and the except will not trigger, or it will be invalid, because 8bit, and so should be set to 8bit just like all the other cases where there's invalid 8bit data. So I think this patch should just be reverted. Can someone (Steve Turnbull?) confirm or refute my analysis? -- R. David Murray www.bitdance.com From martin at v.loewis.de Wed Apr 7 01:51:36 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Apr 2010 01:51:36 +0200 Subject: [Python-Dev] stabilizing for a release In-Reply-To: References: Message-ID: <4BBBC908.90400@v.loewis.de> Benjamin Peterson wrote: > I propose that we freeze the trunk except for fixes for the buildbots. Thoughts? Freezing sounds fine with me. Martin From steve at pearwood.info Wed Apr 7 02:02:50 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Apr 2010 11:02:50 +1100 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: References: Message-ID: <201004071002.51586.steve@pearwood.info> On Tue, 6 Apr 2010 04:25:08 pm Cesare Di Mauro wrote: > It will certainly. There's MUCH that can be optimized to let CPython > squeeze more performance from static analysis (even a gross one) on > locals. [...] > They are just "dummy" examples, but can make it clear how far > optimizations can go with static analysis on locals. Python is a > language that make it possible to use such analysis at compile time, > and I think it is a very good thing. I'm not opposed to the idea of optimisations in general (far from it!) but in case anyone is thinking about doing any work in this area, please be careful about floating point optimisations. E.g. given a float x, you can't assume that x*0 == 0. Nor can you assume that 0-x is the same as -x. (The second is *almost* always correct, except for one float value.) See, for example, the various writings by Professor Kahan: http://www.drdobbs.com/184410314 http://www.cs.berkeley.edu/~wkahan/ Most of the issues discussed apply to languages that deal with floats at a lower level than Python does, but still, simple minded optimizations will break corner cases no matter what language you use. -- Steven D'Aprano From martin at v.loewis.de Wed Apr 7 02:18:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Apr 2010 02:18:13 +0200 Subject: [Python-Dev] iso-2022 and issue 7472: question for the experts In-Reply-To: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> References: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> Message-ID: <4BBBCF45.80103@v.loewis.de> > Can someone (Steve Turnbull?) confirm or refute my analysis? Refute, see http://bugs.python.org/issue804885 > ISO-2022 input will > be 7-bit, and the except will not trigger This conclusion is false: 1. it is 7-bit py> unichr(913).encode("iso-2022-jp") '\x1b$B&!\x1b(B' 2. the except *will* trigger, anyway. py> unichr(913).encode("ascii") Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character u'\u0391' in position 0: ordinal not in range(128) This issue doesn't get noticed, because it doesn't hurt today's email readers and MTAs to process 8-bit MIME, even if 7-bit had been sufficient. HTH, Martin P.S. Notice that Tokio's original patch had the spelling right. From benjamin at python.org Wed Apr 7 03:09:52 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 6 Apr 2010 20:09:52 -0500 Subject: [Python-Dev] stabilizing for a release In-Reply-To: <4BBBC908.90400@v.loewis.de> References: <4BBBC908.90400@v.loewis.de> Message-ID: Let's do it. Please no commits to the trunk which are not aimed at fixing the current buildbot failures. Thank you! 2010/4/6 "Martin v. L?wis" : > Benjamin Peterson wrote: >> I propose that we freeze the trunk except for fixes for the buildbots. Thoughts? > > Freezing sounds fine with me. > > Martin > -- Regards, Benjamin From ncoghlan at gmail.com Wed Apr 7 03:33:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 07 Apr 2010 11:33:12 +1000 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBBACD4.4000906@canterbury.ac.nz> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBB22E7.6060209@voidspace.org.uk> <4BBBACD4.4000906@canterbury.ac.nz> Message-ID: <4BBBE0D8.5040200@gmail.com> Greg Ewing wrote: > Michael Foord wrote: > >> *However*, a project that would be interesting - and that I have >> wanted to do in order to program microcontrollers with *very* small >> memory address spaces [1] - would be to compile a static subset of >> Python down to C. > > That would be an excellent project -- if the result were > successful, I'd be interested in using it! I thought RPython already supported this? (admittedly, my knowledge of of the inner workings of PyPy is fairly sketchy, but I thought static compilation of RPython to a variety of backend targets was one of the key building blocks) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Wed Apr 7 03:40:31 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Apr 2010 21:40:31 -0400 Subject: [Python-Dev] PEP 3147 working implementation In-Reply-To: <20100401161242.6405053d@heresy> References: <20100401161242.6405053d@heresy> Message-ID: <20100406214031.5f5143ec@heresy> On Apr 01, 2010, at 04:12 PM, Barry Warsaw wrote: >I now have a working implementation of PEP 3147 which passes all the existing, >and new, tests. I'm sure there's still work to do, but I think the branch >is in good enough shape to start getting some feedback from python-dev. Thanks to Antoine for doing a review of the diff on Rietveld. I've uploaded a second patch set that addresses these and other issues. I think there are still a few corners of PEP 3147 not yet implemented, and I need to update some documentation, but I think it's getting close enough for consideration soon. The Rietveld issue is here: http://codereview.appspot.com/842043/show Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From rdmurray at bitdance.com Wed Apr 7 03:56:21 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 06 Apr 2010 21:56:21 -0400 Subject: [Python-Dev] iso-2022 and issue 7472: question for the experts In-Reply-To: <4BBBCF45.80103@v.loewis.de> References: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> <4BBBCF45.80103@v.loewis.de> Message-ID: <20100407015621.176541FF60B@kimball.webabinitio.net> On Wed, 07 Apr 2010 02:18:13 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > > Can someone (Steve Turnbull?) confirm or refute my analysis? > > Refute, see http://bugs.python.org/issue804885 > > > ISO-2022 input will > > be 7-bit, and the except will not trigger > > This conclusion is false: > > 1. it is 7-bit > > py> unichr(913).encode("iso-2022-jp") > '\x1b$B&!\x1b(B' > > 2. the except *will* trigger, anyway. > > py> unichr(913).encode("ascii") > Traceback (most recent call last): > File "", line 1, in > UnicodeEncodeError: 'ascii' codec can't encode character u'\u0391' in > position 0: ordinal not in range(128) My understanding, however, is that what comes out of get_payload is always a string, not unicode. That is, it would have to be already encoded, and the encode('ascii') trick is just to see if there are any 8 bit bytes. Tracing the code a little farther, though, I now understand that the *input* encoding that the payload is in (which will on output be encoded as iso-2022-xx) can be an eight bit encoding. So, now I understand the patch, and will fix the spelling mistake. Thanks. -- R. David Murray www.bitdance.com From benjamin at python.org Wed Apr 7 04:57:59 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 6 Apr 2010 21:57:59 -0500 Subject: [Python-Dev] PEP 3147 working implementation In-Reply-To: <20100406214031.5f5143ec@heresy> References: <20100401161242.6405053d@heresy> <20100406214031.5f5143ec@heresy> Message-ID: 2010/4/6 Barry Warsaw : > On Apr 01, 2010, at 04:12 PM, Barry Warsaw wrote: > >>I now have a working implementation of PEP 3147 which passes all the existing, >>and new, tests. ?I'm sure there's still work to do, but I think the branch >>is in good enough shape to start getting some feedback from python-dev. > > Thanks to Antoine for doing a review of the diff on Rietveld. ?I've uploaded a > second patch set that addresses these and other issues. > > I think there are still a few corners of PEP 3147 not yet implemented, and I > need to update some documentation, but I think it's getting close enough for > consideration soon. ?The Rietveld issue is here: > > http://codereview.appspot.com/842043/show I've now added a review, too. -- Regards, Benjamin From techtonik at gmail.com Wed Apr 7 12:30:20 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 7 Apr 2010 13:30:20 +0300 Subject: [Python-Dev] stabilizing for a release In-Reply-To: References: <4BBBC908.90400@v.loewis.de> Message-ID: There is still a serious regression in zipfile module: http://bugs.python.org/issue6090 And I would really like to see my issue with difflib tabs committed: =/ http://bugs.python.org/issue7585 -- anatoly t. On Wed, Apr 7, 2010 at 4:09 AM, Benjamin Peterson wrote: > Let's do it. Please no commits to the trunk which are not aimed at > fixing the current buildbot failures. Thank you! > > 2010/4/6 "Martin v. L?wis" : >> Benjamin Peterson wrote: >>> I propose that we freeze the trunk except for fixes for the buildbots. Thoughts? >> >> Freezing sounds fine with me. >> >> Martin >> > > > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/techtonik%40gmail.com > From martin at v.loewis.de Wed Apr 7 13:10:11 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Apr 2010 13:10:11 +0200 Subject: [Python-Dev] stabilizing for a release In-Reply-To: References: <4BBBC908.90400@v.loewis.de> Message-ID: <4BBC6813.70009@v.loewis.de> anatoly techtonik wrote: > There is still a serious regression in zipfile module: > http://bugs.python.org/issue6090 > > And I would really like to see my issue with difflib tabs committed: =/ > http://bugs.python.org/issue7585 None of these are buildbot failures, so they can't go into 2.7b1, by the decision of the release manager. Please accept that. Regards, Martin From techtonik at gmail.com Wed Apr 7 13:49:14 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 7 Apr 2010 14:49:14 +0300 Subject: [Python-Dev] stabilizing for a release In-Reply-To: <4BBC6813.70009@v.loewis.de> References: <4BBBC908.90400@v.loewis.de> <4BBC6813.70009@v.loewis.de> Message-ID: On Wed, Apr 7, 2010 at 2:10 PM, "Martin v. L?wis" wrote: > >> There is still a serious regression in zipfile module: >> http://bugs.python.org/issue6090 >> >> And I would really like to see my issue with difflib tabs committed: =/ >> http://bugs.python.org/issue7585 > > None of these are buildbot failures, so they can't go into 2.7b1, > by the decision of the release manager. Please accept that. And where will they go? -- anatoly t. From fuzzyman at voidspace.org.uk Wed Apr 7 13:54:51 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 07 Apr 2010 12:54:51 +0100 Subject: [Python-Dev] stabilizing for a release In-Reply-To: References: <4BBBC908.90400@v.loewis.de> Message-ID: <4BBC728B.1070808@voidspace.org.uk> On 07/04/2010 11:30, anatoly techtonik wrote: > There is still a serious regression in zipfile module: > http://bugs.python.org/issue6090 > > And I would really like to see my issue with difflib tabs committed: =/ > http://bugs.python.org/issue7585 > > The zipfile issue looks like it could be fixed for beta 2. Not so sure about the difflib one. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From martin at v.loewis.de Wed Apr 7 13:58:35 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Apr 2010 13:58:35 +0200 Subject: [Python-Dev] stabilizing for a release In-Reply-To: References: <4BBBC908.90400@v.loewis.de> <4BBC6813.70009@v.loewis.de> Message-ID: <4BBC736B.4050408@v.loewis.de> anatoly techtonik wrote: > On Wed, Apr 7, 2010 at 2:10 PM, "Martin v. L?wis" wrote: >>> There is still a serious regression in zipfile module: >>> http://bugs.python.org/issue6090 >>> >>> And I would really like to see my issue with difflib tabs committed: =/ >>> http://bugs.python.org/issue7585 >> None of these are buildbot failures, so they can't go into 2.7b1, >> by the decision of the release manager. Please accept that. > > And where will they go? For the former, a patch is still needed. For the latter, David will decide. Regards, Martin From asmodai at in-nomine.org Wed Apr 7 14:39:28 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 7 Apr 2010 14:39:28 +0200 Subject: [Python-Dev] ffi junk messages Message-ID: <20100407123928.GH48953@nexus.in-nomine.org> Before I file a bug report, is anyone else seeing this (in my case on FreeBSD 8): Modules/_ctypes/libffi/src/x86/sysv.S:360: Error: junk at end of line, first unrecognized character is `@' Modules/_ctypes/libffi/src/x86/sysv.S:387: Error: junk at end of line, first unrecognized character is `@' Modules/_ctypes/libffi/src/x86/sysv.S:423: Error: junk at end of line, first unrecognized character is `@' -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Tattva, achintya bheda abheda tattva... From dickinsm at gmail.com Wed Apr 7 15:04:11 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 7 Apr 2010 14:04:11 +0100 Subject: [Python-Dev] ffi junk messages In-Reply-To: <20100407123928.GH48953@nexus.in-nomine.org> References: <20100407123928.GH48953@nexus.in-nomine.org> Message-ID: On Wed, Apr 7, 2010 at 1:39 PM, Jeroen Ruigrok van der Werven wrote: > Before I file a bug report, is anyone else seeing this (in my case on > FreeBSD 8): > > Modules/_ctypes/libffi/src/x86/sysv.S:360: Error: junk at end of line, first unrecognized character is `@' > Modules/_ctypes/libffi/src/x86/sysv.S:387: Error: junk at end of line, first unrecognized character is `@' > Modules/_ctypes/libffi/src/x86/sysv.S:423: Error: junk at end of line, first unrecognized character is `@' It's on the buildbots, too. See: http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%20trunk/builds/208/steps/compile/logs/stdio Mark From martin at v.loewis.de Wed Apr 7 15:29:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Apr 2010 15:29:14 +0200 Subject: [Python-Dev] ffi junk messages In-Reply-To: References: <20100407123928.GH48953@nexus.in-nomine.org> Message-ID: <4BBC88AA.80805@v.loewis.de> Mark Dickinson wrote: > On Wed, Apr 7, 2010 at 1:39 PM, Jeroen Ruigrok van der Werven > wrote: >> Before I file a bug report, is anyone else seeing this (in my case on >> FreeBSD 8): >> >> Modules/_ctypes/libffi/src/x86/sysv.S:360: Error: junk at end of line, first unrecognized character is `@' >> Modules/_ctypes/libffi/src/x86/sysv.S:387: Error: junk at end of line, first unrecognized character is `@' >> Modules/_ctypes/libffi/src/x86/sysv.S:423: Error: junk at end of line, first unrecognized character is `@' > > It's on the buildbots, too. See: > > http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%20trunk/builds/208/steps/compile/logs/stdio Instead of submitting a bug report, it would be better to submit a patch, though. Can you try having the build process use freebsd.S instead of sysv.S? Regards, Martin From asmodai at in-nomine.org Wed Apr 7 15:30:26 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 7 Apr 2010 15:30:26 +0200 Subject: [Python-Dev] ffi junk messages In-Reply-To: <4BBC88AA.80805@v.loewis.de> References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> Message-ID: <20100407133026.GJ48953@nexus.in-nomine.org> -On [20100407 15:29], "Martin v. L?wis" (martin at v.loewis.de) wrote: >Instead of submitting a bug report, it would be better to submit a >patch, though. Can you try having the build process use freebsd.S >instead of sysv.S? Mark and me are looking at it right now. I can compile ctypes using --with-system-ffi at least. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Grabbing at angels when I fall, for all my demons to applaud, I am free... From ziade.tarek at gmail.com Wed Apr 7 16:01:35 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 7 Apr 2010 16:01:35 +0200 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: <20100401233652.764823A40AF@sparrow.telecommunity.com> References: <20100401233652.764823A40AF@sparrow.telecommunity.com> Message-ID: 2010/4/2 P.J. Eby : [...] > > * Paths under the base installation location are relative to the base > * Paths not under the base installation location, but under the installation > prefix, are also stored relative to the base, IF the base location is a > subpath of the installation prefix > * All other paths are absolute. > > Where "base location" is the effective --install-lib directory, and prefix > is the effective --prefix. ?(Which default of course to site-package and > sys.prefix respectively, but the spec shouldn't be in terms of those > defaults.) Just to make sure we agree on this: we use relative path if the file is in site-packages, or somewhere under sys.prefix. For the latter this is only if site-packages is under sys.prefix. Examples under debian: docutils/__init__.py -> located in /usr/local/lib/python2.6/site-packages/ ../../../bin/rst2html.py -> located in /usr/local/bin /etc/whatever -> located in /etc So, everything under /usr/local (sys.prefix) is relative to /usr/local/lib/python2.6/site-packages. Other paths are absolute. In case the site-packages directory was not under sys.prefix, we would use an absolute path for files under sys.prefix but not in site-packages. (like rst2html.py) Regards Tarek -- Tarek Ziad? | http://ziade.org From exarkun at twistedmatrix.com Wed Apr 7 16:11:41 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 07 Apr 2010 14:11:41 -0000 Subject: [Python-Dev] ffi junk messages In-Reply-To: <4BBC88AA.80805@v.loewis.de> References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> Message-ID: <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> On 01:29 pm, martin at v.loewis.de wrote: >Mark Dickinson wrote: >>On Wed, Apr 7, 2010 at 1:39 PM, Jeroen Ruigrok van der Werven >> wrote: >>>Before I file a bug report, is anyone else seeing this (in my case on >>>FreeBSD 8): >>> >>>Modules/_ctypes/libffi/src/x86/sysv.S:360: Error: junk at end of >>>line, first unrecognized character is `@' >>>Modules/_ctypes/libffi/src/x86/sysv.S:387: Error: junk at end of >>>line, first unrecognized character is `@' >>>Modules/_ctypes/libffi/src/x86/sysv.S:423: Error: junk at end of >>>line, first unrecognized character is `@' >> >>It's on the buildbots, too. See: >> >>http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%20trunk/builds/208/steps/compile/logs/stdio > >Instead of submitting a bug report, it would be better to submit a In *addition* to submitted a bug report, surely. :) >patch, though. Can you try having the build process use freebsd.S >instead of sysv.S? > >Regards, >Martin >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python- >dev/exarkun%40twistedmatrix.com From martin at v.loewis.de Wed Apr 7 16:17:11 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Apr 2010 16:17:11 +0200 Subject: [Python-Dev] ffi junk messages In-Reply-To: <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> Message-ID: <4BBC93E7.3060702@v.loewis.de> >> Instead of submitting a bug report, it would be better to submit a > > In *addition* to submitted a bug report, surely. :) I'm not so sure. It's a ctypes/libffi bug, so most likely, nobody will be able to fix it when reported. For platform-specific libffi bugs, the patch most likely will come from the submitter, as nobody else might have access to the platform, and the knowledge about the platform details. Regards, Martin From asmodai at in-nomine.org Wed Apr 7 16:33:20 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 7 Apr 2010 16:33:20 +0200 Subject: [Python-Dev] ffi junk messages In-Reply-To: <4BBC93E7.3060702@v.loewis.de> References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> <4BBC93E7.3060702@v.loewis.de> Message-ID: <20100407143320.GK48953@nexus.in-nomine.org> -On [20100407 16:17], "Martin v. L?wis" (martin at v.loewis.de) wrote: >I'm not so sure. It's a ctypes/libffi bug, so most likely, nobody will >be able to fix it when reported. For platform-specific libffi bugs, the >patch most likely will come from the submitter, as nobody else might >have access to the platform, and the knowledge about the platform details. No it is not actually. Mark and me found the culprit in the Python tree. Commit incoming. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B I dream of Love as Time runs through my hand... From pje at telecommunity.com Wed Apr 7 16:35:24 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Apr 2010 10:35:24 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> Message-ID: <20100407143530.D68913A4063@sparrow.telecommunity.com> At 04:01 PM 4/7/2010 +0200, Tarek Ziad? wrote: >2010/4/2 P.J. Eby : >[...] > > > > * Paths under the base installation location are relative to the base > > * Paths not under the base installation location, but under the > installation > > prefix, are also stored relative to the base, IF the base location is a > > subpath of the installation prefix > > * All other paths are absolute. > > > > Where "base location" is the effective --install-lib directory, and prefix > > is the effective --prefix. (Which default of course to site-package and > > sys.prefix respectively, but the spec shouldn't be in terms of those > > defaults.) > >Just to make sure we agree on this: > >we use relative path if the file is in site-packages, or somewhere >under sys.prefix. For the latter this is only if site-packages is >under sys.prefix. Um, sys.prefix, or the prefix set by "setup.py install --prefix" (which of course defaults to sys.prefix)? From ziade.tarek at gmail.com Wed Apr 7 16:40:35 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 7 Apr 2010 16:40:35 +0200 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: <20100407143530.D68913A4063@sparrow.telecommunity.com> References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> Message-ID: On Wed, Apr 7, 2010 at 4:35 PM, P.J. Eby wrote: > At 04:01 PM 4/7/2010 +0200, Tarek Ziad? wrote: >> >> 2010/4/2 P.J. Eby : >> [...] >> > >> > * Paths under the base installation location are relative to the base >> > * Paths not under the base installation location, but under the >> > installation >> > prefix, are also stored relative to the base, IF the base location is a >> > subpath of the installation prefix >> > * All other paths are absolute. >> > >> > Where "base location" is the effective --install-lib directory, and >> > prefix >> > is the effective --prefix. ?(Which default of course to site-package and >> > sys.prefix respectively, but the spec shouldn't be in terms of those >> > defaults.) >> >> Just to make sure we agree on this: >> >> we use relative path if the file is in site-packages, or somewhere >> under sys.prefix. For the latter this is only if site-packages is >> under sys.prefix. > > Um, sys.prefix, or the prefix set by "setup.py install --prefix" (which of > course defaults to sys.prefix)? Yes, I used default values to make the text clearer. so for the PEP : - sys.prefix -> the installation prefix provided by --prefix at installation time - site-packages -> the installation libdir, provided by --install-lib at installation time -- Tarek Ziad? | http://ziade.org From solipsis at pitrou.net Wed Apr 7 16:48:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Apr 2010 14:48:12 +0000 (UTC) Subject: [Python-Dev] ffi junk messages References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> <4BBC93E7.3060702@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > >> Instead of submitting a bug report, it would be better to submit a > > > > In *addition* to submitted a bug report, surely. :) > > I'm not so sure. It's a ctypes/libffi bug, so most likely, nobody will > be able to fix it when reported. It's probably still good to record the existence of these bugs, even if we don't know how to fix them. That way, other people which get the bug can contribute to the issue, and perhaps work together towards resolving it. (also, as I understand it, the latest ctypes issues seem to have popped up after an update of the bundled libffi, so perhaps that update wasn't totally right, didn't choose the right libffi version, or missed some files?) Regards Antoine. From michael at voidspace.org.uk Wed Apr 7 16:54:52 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 07 Apr 2010 15:54:52 +0100 Subject: [Python-Dev] Episode 11 of A Little Bit of Python Message-ID: <4BBC9CBC.8010000@voidspace.org.uk> Hello all, Not *strictly* on topic, but probably of interest nonetheless (so my apologies). Episode 11 of "A Little Bit of Python" is now available. An interview with core-CPython developer Antoine Pitrou. http://advocacy.python.org/podcasts/littlebit/2010-04-07.mp3 Many thanks to Antoine for the interview. All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From martin at v.loewis.de Wed Apr 7 17:29:45 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 07 Apr 2010 17:29:45 +0200 Subject: [Python-Dev] ffi junk messages In-Reply-To: References: <20100407123928.GH48953@nexus.in-nomine.org> <4BBC88AA.80805@v.loewis.de> <20100407141141.2779.488946084.divmod.xquotient.148@localhost.localdomain> <4BBC93E7.3060702@v.loewis.de> Message-ID: <4BBCA4E9.4050205@v.loewis.de> > (also, as I understand it, the latest ctypes issues seem to have popped up after > an update of the bundled libffi, so perhaps that update wasn't totally right, > didn't choose the right libffi version, or missed some files?) In the case of the SPARC issue: the bug is still exists in the libffi hg repository. Regards, Martin From ianb at colorstudy.com Wed Apr 7 18:33:04 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 7 Apr 2010 11:33:04 -0500 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> Message-ID: On Wed, Apr 7, 2010 at 9:40 AM, Tarek Ziad? wrote: > so for the PEP : > > - sys.prefix -> the installation prefix provided by --prefix at > installation time > - site-packages -> the installation libdir, provided by --install-lib > at installation time How do you actually calculate site-packages? Would you store the directory name somewhere? Would you import the module and look at os.path.dirname(os.path.dirname(module.__file__))? Or just scan to see where the module would be? If you store the directory name somewhere then you have another absolute path. This is why, for simplicity, I thought it should be relative to the directory where the record file is (lots of extraneous ../, but the most obvious meaning of a relative filename). -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Apr 7 19:40:35 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Apr 2010 13:40:35 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> Message-ID: <20100407174042.597E83A4063@sparrow.telecommunity.com> At 12:48 AM 4/4/2010 +0200, Tarek Ziad? wrote: >The implementation so far will load zip files founded in the paths, >see ZippedDistributionDir/ZippedDistribution. I was saying that it doesn't support sys.path entries of the form: "some/path/to/somezipfile.zip/subdir1" Python works correctly for importing with this, but the ZipFinder class in the implementation throws away 'subdir1' and only scans the root of the zipfile, silently generating incorrect results in such a case. To fix the problem, you would need to make use of the .archive and .prefix attributes of defined by the underlying zipimporter class. After you call the super().__init__ method, .archive points to the actual zipfile path, and .prefix contains the portion of the path that's inside the zipfile. You can then just adjust your path usage in ZipFinder and ZippedDistribution accordingly. Also, as far as I can tell from the source, ZipFinder instances are never actually created, except through explicit usage in some of the tests. i.e., it doesn't work out of the box right now. However, this could be fixed with the addition of the code snippets below... >I am wondering though, if we shouldn't omit this zip story for PEP 376 >altogether, and work later on >something more generic wrt import protocols where modules/packages >could be stored anywhere. Essentially, you could do something like: @simplegeneric # <-already in stdlib pkgutil since 2.5 def dist_finder_for(importer): if hasattr(importer, 'list_distributions'): return importer return None dist_finder_for.register(ImpImporter, FSFinder) dist_finder_for.register(zipimporter, ZipFinder) And then your all_finders() function would be reduced to: def all_finders(): for importer in iter_importers(): # <-already in stdlib pkgutil finder = dist_finder_for(importer) if finder is not None: yield finder As you can see, it's pretty easy to integrate with your existing code. simplegeneric and iter_importers (as well as a get_importer(pathentry) function, ImpImporter, etc.) have been in pkgutil since 2.5, but you can always backport them if you're making a standalone version for 2.4. From pje at telecommunity.com Wed Apr 7 19:45:20 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Apr 2010 13:45:20 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> Message-ID: <20100407174526.4472D3A4063@sparrow.telecommunity.com> At 04:40 PM 4/7/2010 +0200, Tarek Ziad? wrote: >On Wed, Apr 7, 2010 at 4:35 PM, P.J. Eby wrote: > > At 04:01 PM 4/7/2010 +0200, Tarek Ziad? wrote: > >> > >> 2010/4/2 P.J. Eby : > >> [...] > >> > > >> > * Paths under the base installation location are relative to the base > >> > * Paths not under the base installation location, but under the > >> > installation > >> > prefix, are also stored relative to the base, IF the base location is a > >> > subpath of the installation prefix > >> > * All other paths are absolute. > >> > > >> > Where "base location" is the effective --install-lib directory, and > >> > prefix > >> > is the effective --prefix. (Which default of course to site-package and > >> > sys.prefix respectively, but the spec shouldn't be in terms of those > >> > defaults.) > >> > >> Just to make sure we agree on this: > >> > >> we use relative path if the file is in site-packages, or somewhere > >> under sys.prefix. For the latter this is only if site-packages is > >> under sys.prefix. > > > > Um, sys.prefix, or the prefix set by "setup.py install --prefix" (which of > > course defaults to sys.prefix)? > >Yes, I used default values to make the text clearer. > >so for the PEP : > >- sys.prefix -> the installation prefix provided by --prefix at >installation time >- site-packages -> the installation libdir, provided by --install-lib >at installation time Yes. Looking more closely at your example, though: >Examples under debian: > > docutils/__init__.py -> located in >/usr/local/lib/python2.6/site-packages/ > ../../../bin/rst2html.py -> located in /usr/local/bin > /etc/whatever -> located in /etc I'm wondering if there's really any benefit to having ../../../bin/rst2html.py vs. /usr/local/bin/rst2html.py. Was there a use case for that, or should we just go with relative paths ONLY for children of the libdir? (I only suggested this setup in order to preserve as much of the prefix-relativity proposal as possible, but I wasn't the one who proposed prefix-relativity so I don't recall what the use case is, and I don't even remember who proposed it. I only ever had a usecase for libdir-relativity personally.) From pje at telecommunity.com Wed Apr 7 19:48:38 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Apr 2010 13:48:38 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> Message-ID: <20100407174844.E83A43A4063@sparrow.telecommunity.com> At 11:33 AM 4/7/2010 -0500, Ian Bicking wrote: >On Wed, Apr 7, 2010 at 9:40 AM, Tarek Ziad?? ><ziade.tarek at gmail.com> wrote: >so for the PEP : > >- sys.prefix -> the installation prefix provided by --prefix at >installation time >- site-packages -> the installation libdir, provided by --install-lib >at installation time > > >How do you actually calculate site-packages? ? Would you store the >directory name somewhere? ? Would you import the module and look at >os.path.dirname(os.path.dirname(module.__file__))? ? Or just scan to >see where the module would be? > >If you store the directory name somewhere then you have another >absolute path. ? This is why, for simplicity, I thought it should be >relative to the directory where the record file is (lots of >extraneous ../, but the most obvious meaning of a relative filename). The paths are relative to the sys.path entry for which the metadata applies - that's how you get to finding the .dist-info in the first place, so that's what you os.path.join to the paths in the record. There's no "storing" of the directory name needed. From ianb at colorstudy.com Wed Apr 7 19:51:09 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 7 Apr 2010 12:51:09 -0500 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: <20100407174526.4472D3A4063@sparrow.telecommunity.com> References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> <20100407174526.4472D3A4063@sparrow.telecommunity.com> Message-ID: On Wed, Apr 7, 2010 at 12:45 PM, P.J. Eby wrote: > Examples under debian: >> >> docutils/__init__.py -> located in >> /usr/local/lib/python2.6/site-packages/ >> ../../../bin/rst2html.py -> located in /usr/local/bin >> /etc/whatever -> located in /etc >> > > I'm wondering if there's really any benefit to having > ../../../bin/rst2html.py vs. /usr/local/bin/rst2html.py. Was there a use > case for that, or should we just go with relative paths ONLY for children of > the libdir? > > (I only suggested this setup in order to preserve as much of the > prefix-relativity proposal as possible, but I wasn't the one who proposed > prefix-relativity so I don't recall what the use case is, and I don't even > remember who proposed it. I only ever had a usecase for libdir-relativity > personally.) Yes, in a virtualenv environment there will be ../../../bin/rst2html.py that will still be under the (virtual) sys.prefix, and the whole bundle can be usefully moved around. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Apr 7 19:56:46 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Apr 2010 13:56:46 -0400 Subject: [Python-Dev] Proposing PEP 376 In-Reply-To: References: <20100401233652.764823A40AF@sparrow.telecommunity.com> <20100407143530.D68913A4063@sparrow.telecommunity.com> <20100407174526.4472D3A4063@sparrow.telecommunity.com> Message-ID: <20100407175651.9AB9A3A4063@sparrow.telecommunity.com> At 12:51 PM 4/7/2010 -0500, Ian Bicking wrote: >On Wed, Apr 7, 2010 at 12:45 PM, P.J. Eby ><pje at telecommunity.com> wrote: >Examples under debian: > >? ? docutils/__init__.py ? ? ? ? ? -> ? ? located in >/usr/local/lib/python2.6/site-packages/ >? ? ../../../bin/rst2html.py ? ? ? -> ? located in /usr/local/bin >? ? /etc/whatever ? ? ? ? ? ? ? ? ? -> ? ? located in /etc > > >I'm wondering if there's really any benefit to having >../../../bin/rst2html.py vs. /usr/local/bin/rst2html.py. ? Was there >a use case for that, or should we just go with relative paths ONLY >for children of the libdir? > >(I only suggested this setup in order to preserve as much of the >prefix-relativity proposal as possible, but I wasn't the one who >proposed prefix-relativity so I don't recall what the use case is, >and I don't even remember who proposed it. ? I only ever had a >usecase for libdir-relativity personally.) > > >Yes, in a virtualenv environment there will be >../../../bin/rst2html.py that will still be under the (virtual) >sys.prefix, and the whole bundle can be usefully moved around. Ah, ok. Good! +1, then. From rdmurray at bitdance.com Wed Apr 7 20:52:00 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 07 Apr 2010 14:52:00 -0400 Subject: [Python-Dev] [python-committers] stabilizing for a release In-Reply-To: <4BBC728B.1070808@voidspace.org.uk> References: <4BBBC908.90400@v.loewis.de> <4BBC728B.1070808@voidspace.org.uk> Message-ID: <20100407185200.5A3121BC343@kimball.webabinitio.net> On Wed, 07 Apr 2010 12:54:51 +0100, Michael Foord wrote: >On 07/04/2010 11:30, anatoly techtonik wrote: >> There is still a serious regression in zipfile module: >> http://bugs.python.org/issue6090 >> >> And I would really like to see my issue with difflib tabs committed: =/ >> http://bugs.python.org/issue7585 >> >> >The zipfile issue looks like it could be fixed for beta 2. Not so sure >about the difflib one. As Antoine pointed out, 7585 is actually a bug fix (making difflib follow the unified diff 'standard', as it was originally specified to do). So unless Benjamin objects I plan to commit it after the freeze. --David From turnbull at sk.tsukuba.ac.jp Wed Apr 7 21:22:02 2010 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 08 Apr 2010 04:22:02 +0900 Subject: [Python-Dev] iso-2022 and issue 7472: question for the experts In-Reply-To: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> References: <20100406231524.55EA11FC0BF@kimball.webabinitio.net> Message-ID: <87zl1fnk4l.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > A long time ago (in a galaxy far far...no, wrong show) > > Er, as I was saying, a long time ago Barry applied a patch to > email that went more or less like this: > > ndex: email/Encoders.py > =================================================================== > --- email/Encoders.py (revision 35918) > +++ email/Encoders.py (revision 35919) > @@ -84,7 +83,13 @@ > try: > orig.encode('ascii') > except UnicodeError: > - msg['Content-Transfer-Encoding'] = '8bit' > + # iso-2022-* is non-ASCII but still 7-bit This comment may be inaccurate. The ISO 2022 family includes what are normally "8bit" encodings such as the EUC family and ISO 8859. I don't know whether there are any IANA-registered 8bit charsets with names that start with 'iso-2022-', and AFAIK there are none in Python. (There is an 'iso-2022-8' encoding in Emacs, though.) Still, I'd be more comfortable with an explicit list than with the .startswith('iso-2022-') idiom. > + charset = msg.get_charset() > + output_cset = charset and charset.output_charset > + if output_cset and output_cset.lower().startswith('iso-2202-'): > + msg['Content-Transfer-Encoding'] = '7bit' > + else: > + msg['Content-Transfer-Encoding'] = '8bit' > else: > msg['Content-Transfer-Encoding'] = '7bit' > Reading the standards, it looks to me like either the ISO-2022 > input will be 7-bit, and the except will not trigger, or it will be > invalid, because 8bit, and so should be set to 8bit just like all > the other cases where there's invalid 8bit data. So I think this > patch should just be reverted. I have nothing to add to what Martin said about the basic analysis. It would be possible to just unconditionally set the Content-Transfer-Encoding to 8bit, although that may violate a SHOULD in the MIME standard. From martin at v.loewis.de Thu Apr 8 01:13:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Apr 2010 01:13:27 +0200 Subject: [Python-Dev] regrtest oddify Message-ID: <4BBD1197.1060709@v.loewis.de> I have commented out all tests in test_gdb, yet http://www.python.org/dev/buildbot/all/builders/sparc%20Ubuntu%20trunk/builds/47/steps/test/logs/stdio still shows them being run. Can anybody explain that, please? TIA, Martin From benjamin at python.org Thu Apr 8 01:20:13 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 7 Apr 2010 18:20:13 -0500 Subject: [Python-Dev] regrtest oddify In-Reply-To: <4BBD1197.1060709@v.loewis.de> References: <4BBD1197.1060709@v.loewis.de> Message-ID: 2010/4/7 "Martin v. L?wis" : > I have commented out all tests in test_gdb, yet > > http://www.python.org/dev/buildbot/all/builders/sparc%20Ubuntu%20trunk/builds/47/steps/test/logs/stdio > > still shows them being run. Can anybody explain that, please? That's because the buildbot only updated to the revision before your change. -- Regards, Benjamin From yaniv at aknin.name Thu Apr 8 08:11:09 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 8 Apr 2010 16:11:09 +1000 Subject: [Python-Dev] Issue #7978, unexpected EINTR-related exceptions Message-ID: Issue #7978 (http://bugs.python.org/issue7978) describes a bug in SocketServer where a received signal in SocketServer's select() call will raise an uncaught exception due to EINTR. The proposed solution was to wrap SocketServer's select() with something like twisted's untilConcludes function, which catches EINTR exceptions and re-calls the call (see issue for code). However, before committing this to SocketServer, neologix raised the (valid) concern that this is generally-useful code, already duplicated at least in subprocess (_eintr_retry_call, subprocess.py), which can probably be moved to another module and be shared from there. It was recommended to bring this issue in the broader context to the list, but from my look at the archives no one did, hence this message. There are a few more points of discussion on the issue's roundup thread which I won't list here, so it's best to look there if you care. Also, I'm rather new on python-dev and I'm not sure how duplicate python-dev/roundup threads are usually handled, but I humbly think if this issue interests you it's best to add your thoughts to the issue's comments, rather than here; please correct me if this is a misguided assumption. Cheers, - Yaniv From victor.stinner at haypocalc.com Thu Apr 8 11:11:16 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 8 Apr 2010 11:11:16 +0200 Subject: [Python-Dev] Issue #7978, unexpected EINTR-related exceptions In-Reply-To: References: Message-ID: <201004081111.16299.victor.stinner@haypocalc.com> Le jeudi 08 avril 2010 08:11:09, Yaniv Aknin a ?crit : > Issue #7978 (http://bugs.python.org/issue7978) describes a bug in > SocketServer where a received signal in SocketServer's select() call > will raise an uncaught exception due to EINTR. The proposed solution > was to wrap SocketServer's select() with something like twisted's > untilConcludes function, which catches EINTR exceptions and re-calls > the call (see issue for code). > > However, before committing this to SocketServer, neologix raised the > (valid) concern that this is generally-useful code, already duplicated > at least in subprocess (_eintr_retry_call, subprocess.py), which can > probably be moved to another module and be shared from there. (...) +1 to share EINTR-related code, but I don't know the best place for such functions. Maybe the os module? -- Victor Stinner http://www.haypocalc.com/ From barry at python.org Thu Apr 8 15:08:15 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Apr 2010 09:08:15 -0400 Subject: [Python-Dev] PEP 3147 working implementation In-Reply-To: References: <20100401161242.6405053d@heresy> <20100406214031.5f5143ec@heresy> Message-ID: <20100408090815.2c7caf40@heresy> On Apr 06, 2010, at 09:57 PM, Benjamin Peterson wrote: >I've now added a review, too. As did Brett. Thanks! I've responded and will upload a new patch set as soon as I've verified the test suite passes. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From cesare.di.mauro at gmail.com Thu Apr 8 15:54:21 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Thu, 8 Apr 2010 16:54:21 +0300 Subject: [Python-Dev] Scope object (Re: nonlocals() function?) In-Reply-To: <201004071002.51586.steve@pearwood.info> References: <201004071002.51586.steve@pearwood.info> Message-ID: 2010/4/7 Steven D'Aprano > On Tue, 6 Apr 2010 04:25:08 pm Cesare Di Mauro wrote: > > > It will certainly. There's MUCH that can be optimized to let CPython > > squeeze more performance from static analysis (even a gross one) on > > locals. > [...] > > They are just "dummy" examples, but can make it clear how far > > optimizations can go with static analysis on locals. Python is a > > language that make it possible to use such analysis at compile time, > > and I think it is a very good thing. > > I'm not opposed to the idea of optimisations in general (far from it!) > but in case anyone is thinking about doing any work in this area, > please be careful about floating point optimisations. E.g. given a float > x, you can't assume that x*0 == 0. Nor can you assume that 0-x is the > same as -x. (The second is *almost* always correct, except for one > float value.) > > See, for example, the various writings by Professor Kahan: > > http://www.drdobbs.com/184410314 > http://www.cs.berkeley.edu/~wkahan/ > > Most of the issues discussed apply to languages that deal with floats at > a lower level than Python does, but still, simple minded optimizations > will break corner cases no matter what language you use. > > -- > Steven D'Aprano Thanks for the useful links. I never applied such kind of optimizations, and I think I'll never to do it anyway. :) Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmalcolm at redhat.com Thu Apr 8 17:56:53 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Thu, 08 Apr 2010 11:56:53 -0400 Subject: [Python-Dev] regrtest oddify In-Reply-To: References: <4BBD1197.1060709@v.loewis.de> Message-ID: <1270742213.3066.3.camel@surprise> On Wed, 2010-04-07 at 18:20 -0500, Benjamin Peterson wrote: > 2010/4/7 "Martin v. L?wis" : > > I have commented out all tests in test_gdb, yet > > > > http://www.python.org/dev/buildbot/all/builders/sparc%20Ubuntu%20trunk/builds/47/steps/test/logs/stdio > > > > still shows them being run. Can anybody explain that, please? > > That's because the buildbot only updated to the revision before your change. FWIW I've attached a patch [1] to http://bugs.python.org/issue8330 which I believe may fix the issues seen in that log. Hope this is helpful Dave [1] http://bugs.python.org/file16808 From martin at v.loewis.de Thu Apr 8 18:52:51 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 08 Apr 2010 18:52:51 +0200 Subject: [Python-Dev] regrtest oddify In-Reply-To: <1270742213.3066.3.camel@surprise> References: <4BBD1197.1060709@v.loewis.de> <1270742213.3066.3.camel@surprise> Message-ID: <4BBE09E3.2000901@v.loewis.de> > FWIW I've attached a patch [1] to http://bugs.python.org/issue8330 which > I believe may fix the issues seen in that log. Thanks! I'll apply them after the beta release (don't want to mess with it before). Regards, Martin From greg.ewing at canterbury.ac.nz Thu Apr 8 23:51:28 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 09 Apr 2010 09:51:28 +1200 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBBE0D8.5040200@gmail.com> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBB22E7.6060209@voidspace.org.uk> <4BBBACD4.4000906@canterbury.ac.nz> <4BBBE0D8.5040200@gmail.com> Message-ID: <4BBE4FE0.8020605@canterbury.ac.nz> Nick Coghlan wrote: > I thought RPython already supported this? (admittedly, my knowledge of > of the inner workings of PyPy is fairly sketchy, but I thought static > compilation of RPython to a variety of backend targets was one of the > key building blocks) Maybe so, but one would still have to create the appropriate backend to target the machine in question. I wouldn't like to rely on a generic C-generating backend to target something very tiny in an effective way. -- Greg From status at bugs.python.org Fri Apr 9 18:07:37 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 9 Apr 2010 18:07:37 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100409160737.D0FCC78130@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-04-02 - 2010-04-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2639 open (+37) / 17538 closed (+30) / 20177 total (+67) Open issues with patches: 1072 Average duration of open issues: 728 days. Median duration of open issues: 489 days. Open Issues Breakdown open 2599 (+37) languishing 7 ( +0) pending 32 ( +0) Issues Created Or Reopened (71) _______________________________ Add environment variable $PYTHONWARNINGS 2010-04-07 http://bugs.python.org/issue7301 reopened pjenvey patch, needs review patch: BaseHTTPServer reinventing rfc822 date formatting 2010-04-09 http://bugs.python.org/issue7370 reopened r.david.murray patch test_py3kwarn fails when running the whole test suite 2010-04-03 http://bugs.python.org/issue7772 reopened mark.dickinson patch Documentation of math.pow, math.hypot, and complex.__abs__ 2010-04-06 http://bugs.python.org/issue7947 reopened mark.dickinson patch i have a doubt with using __init__ and .self and classes 2010-04-02 CLOSED http://bugs.python.org/issue8291 created krishnalolla Incorrect condition test in platform.py 2010-04-02 http://bugs.python.org/issue8292 created akuchling patch, easy HTTPSConnection.close() does not immediately close the connect 2010-04-02 http://bugs.python.org/issue8293 created dandrzejewski Allow Fraction constructor to accept float and decimal instanc 2010-04-02 CLOSED http://bugs.python.org/issue8294 created mark.dickinson patch add unpack_archive to shutil 2010-04-02 http://bugs.python.org/issue8295 created tarek multiprocessing.Pool hangs when issuing KeyboardInterrupt 2010-04-03 http://bugs.python.org/issue8296 created vlasovskikh patch AttributeError message text should include module name 2010-04-03 http://bugs.python.org/issue8297 created cjerdonek patch in what way we have to save tha module? 2010-04-03 CLOSED http://bugs.python.org/issue8298 created krishnalolla Improve GIL in 2.7 2010-04-03 http://bugs.python.org/issue8299 created krisvale patch, patch Allow struct.pack to handle objects with an __index__ method. 2010-04-03 CLOSED http://bugs.python.org/issue8300 created mark.dickinson patch Putting a function in a unittest.TestSuite doesn't work 2010-04-03 http://bugs.python.org/issue8301 created michael.foord SkipTest exception in setUpClass or setUpModule is marked as a 2010-04-03 http://bugs.python.org/issue8302 created michael.foord python -m unittest -h and python -m unittest discover -h messa 2010-04-03 http://bugs.python.org/issue8303 created michael.foord strftime and Unicode characters 2010-04-03 http://bugs.python.org/issue8304 created AndiDog memoview[0] creates an invalid view if ndim != 1 2010-04-03 http://bugs.python.org/issue8305 created haypo ctypes.create_string_buffer should only accept bytes 2010-04-03 http://bugs.python.org/issue8306 created benjamin.peterson test_pep263 failure on OS X 2010-04-03 CLOSED http://bugs.python.org/issue8307 created mark.dickinson raw_bytes.decode('cp932') -- spurious mappings 2010-04-03 CLOSED http://bugs.python.org/issue8308 created sjmachin Sin(x) is Wrong 2010-04-04 CLOSED http://bugs.python.org/issue8309 created derekroconnor dis.dis function skips new-style classes in a module 2010-04-04 CLOSED http://bugs.python.org/issue8310 created dogeen wave module sets data subchunk size incorrectly when writing w 2010-04-04 http://bugs.python.org/issue8311 created Jeff.Pursell Add post/pre hooks for distutils commands 2010-04-04 http://bugs.python.org/issue8312 created tarek message in unittest traceb 2010-04-05 http://bugs.python.org/issue8313 created michael.foord test_ctypes fails in test_ulonglong on sparc buildbots 2010-04-05 CLOSED http://bugs.python.org/issue8314 created r.david.murray patch, buildbot ./python -m unittest test.test_importlib doesn't work 2010-04-05 http://bugs.python.org/issue8315 created barry test_gdb is susceptible to tty width settings 2010-04-05 CLOSED http://bugs.python.org/issue8316 created dmalcolm buildbot test_tarfile fails intermittently on Windows 2010-04-05 CLOSED http://bugs.python.org/issue8317 created jaraco Deprecation of multifile inappropriate or incomplete 2010-04-05 http://bugs.python.org/issue8318 created tseaver HTMLparser does not handle call to handle_data when a tag cont 2010-04-05 http://bugs.python.org/issue8319 created wplappert easy docs on socket.recv_into doesn't mention the return value 2010-04-05 CLOSED http://bugs.python.org/issue8320 created irmen easy Give access to openssl version number 2010-04-05 CLOSED http://bugs.python.org/issue8321 created pitrou patch test_ssl failures with OpenSSL 1.0.0 2010-04-05 http://bugs.python.org/issue8322 created pitrou multiprocessing.Queue ignores pickle restrictions in .put() 2010-04-05 http://bugs.python.org/issue8323 created rschoon add a distutils test command 2010-04-05 http://bugs.python.org/issue8324 created tarek gsoc improve regrtest command line help 2010-04-06 http://bugs.python.org/issue8325 created techtonik patch, easy, needs review Cannot import name SemLock on Ubuntu lucid 2010-04-06 http://bugs.python.org/issue8326 created coconutrb unintuitive behaviour of logging message propagation 2010-04-06 CLOSED http://bugs.python.org/issue8327 created pakal longobject.c: silence warnings (Visual Studio) 2010-04-06 CLOSED http://bugs.python.org/issue8328 created skrah patch select.select() can return lists with identical id()'s 2010-04-06 CLOSED http://bugs.python.org/issue8329 created mrmakent Failures seen in test_gdb on buildbots 2010-04-07 http://bugs.python.org/issue8330 created dmalcolm patch, buildbot a documentation grammar fix in logging module 2010-04-07 CLOSED http://bugs.python.org/issue8331 created l0nwlf patch regrtest single TestClass/test_method 2010-04-07 http://bugs.python.org/issue8332 created techtonik test_multiprocessing: pickling failures 2010-04-07 CLOSED http://bugs.python.org/issue8333 created skrah patch, buildbot winreg.QueryValue should return bytes, not unicode 2010-04-07 CLOSED http://bugs.python.org/issue8334 created stutzbach distutils test_build_ext's test_get_outputs fails in bootstrap 2010-04-07 http://bugs.python.org/issue8335 created matejcik PyObject_CallObject - Not "reference-count-neutral" 2010-04-07 CLOSED http://bugs.python.org/issue8336 created Krauzi test_gdb fails on Sparc Ubuntu 2010-04-07 CLOSED http://bugs.python.org/issue8337 created loewis buildbot Outdated information 2010-04-07 http://bugs.python.org/issue8338 created LambertDW patch urlunparse(urlparse('x://')) now returns 'x:' instead of 'x:// 2010-04-07 http://bugs.python.org/issue8339 created Michael Glassford easy bytearray undocumented on trunk 2010-04-07 http://bugs.python.org/issue8340 created brian.curtin sphinx bug? 2010-04-08 CLOSED http://bugs.python.org/issue8341 created LambertDW Python fails to run if launched from NTFS symlink and DLL not 2010-04-08 CLOSED http://bugs.python.org/issue8342 created jaraco patch improve re parse error messages for named groups 2010-04-08 http://bugs.python.org/issue8343 created techtonik patch test_tag_configure fails on FreeBSD 2010-04-08 http://bugs.python.org/issue8344 created loewis buildbot missing method MatchObject.groups in re module doc 2010-04-08 CLOSED http://bugs.python.org/issue8345 created drukker Old Version Name in Interactive Mode 2010-04-08 http://bugs.python.org/issue8346 created istihza string capitalize erroneously lower-case any non-first letters 2010-04-08 CLOSED http://bugs.python.org/issue8347 created famart test_urllib2net fails because of bad URL 2010-04-08 CLOSED http://bugs.python.org/issue8348 created loewis buildbot os.environ.get returns incorrect data 2010-04-08 CLOSED http://bugs.python.org/issue8349 created dbrandow os.mkdir doc comment is incorrect 2010-04-08 http://bugs.python.org/issue8350 created twhitema patch Suppress large diffs in unitttest.TestCase.assertSequenceEqual 2010-04-08 http://bugs.python.org/issue8351 created flub patch imp.find_module of a .py ending dir causes glibc double free c 2010-04-09 CLOSED http://bugs.python.org/issue8352 created doko Negative exponentiation behaving oddly in python shell 2010-04-09 CLOSED http://bugs.python.org/issue8353 created CWardUSC siginterrupt with flag=False is reset when signal received 2010-04-09 http://bugs.python.org/issue8354 created spiv diff.py produce unified format by default 2010-04-09 http://bugs.python.org/issue8355 created techtonik patch SyntaxError: integer assignment with leading zeros (only 8 and 2010-04-09 CLOSED http://bugs.python.org/issue8356 created posativ distutils does not allow installation with --root and --prefix 2010-04-09 http://bugs.python.org/issue8357 created timkersten Issues Now Closed (63) ______________________ float compared to decimal is silently incorrect. 140 days http://bugs.python.org/issue2531 mark.dickinson patch Py_(X)SETREF macros 666 days http://bugs.python.org/issue3081 rhettinger patch Ill-formed surrogates not treated as errors during encoding/de 591 days http://bugs.python.org/issue3672 ezio.melotti patch "sort" command doesn't work in pstats if run interactively 492 days http://bugs.python.org/issue4440 akuchling patch, needs review struct: per item endianess specification 492 days http://bugs.python.org/issue4484 rhettinger Faster utf-8 decoding 451 days http://bugs.python.org/issue4868 ezio.melotti patch Strange DeprecationWarning behaviour in module struct 420 days http://bugs.python.org/issue5198 mark.dickinson test_distutils leaves a 'foo' file behind in the cwd 328 days http://bugs.python.org/issue6022 matejcik warnings.catch_warnings is not thread-safe 241 days http://bugs.python.org/issue6647 akuchling patch decimal.py: == and != comparisons involving NaNs 11 days http://bugs.python.org/issue7279 mark.dickinson patch Add lossy queue to queue library module 137 days http://bugs.python.org/issue7337 rhettinger Code in xrange documentation does not work 79 days http://bugs.python.org/issue7721 georg.brandl Doc for itertools recipe consume is complicated and less effic 68 days http://bugs.python.org/issue7764 rhettinger Revise generator-related Glossary entries 37 days http://bugs.python.org/issue8012 jjposner patch Setting a T_INT attribute raises internal error 41 days http://bugs.python.org/issue8014 mark.dickinson patch Add gdb7 hooks to make it easier to debug Python 34 days http://bugs.python.org/issue8032 loewis patch pybsddb 4.8.4 integration 24 days http://bugs.python.org/issue8156 jcea test_zlib fails with zlib 1.2.4 16 days http://bugs.python.org/issue8193 valerio.turturici patch, easy test_ttk_guionly assertion error on 3.x linux 64-bit 17 days http://bugs.python.org/issue8204 loewis patch, buildbot test_issue7820 fails: "name '?' is not defined" 12 days http://bugs.python.org/issue8208 ezio.melotti Fix typos and phrasing in the Web servers howto 12 days http://bugs.python.org/issue8218 ezio.melotti patch, easy, needs review Fix C API documentation: Argument parsing 9 days http://bugs.python.org/issue8227 haypo patch Decimal constructor to accept float 9 days http://bugs.python.org/issue8257 mark.dickinson patch, easy left shift operator doesn't accepts long as second argument 8 days http://bugs.python.org/issue8259 mark.dickinson patch bad wording in error message attempting to start a Thread twic 7 days http://bugs.python.org/issue8262 orsenthil patch python-gdb.py triggers compile errors on FreeBSD and Solaris 3 days http://bugs.python.org/issue8287 r.david.murray patch, buildbot zipfile module documentation misprint 1 days http://bugs.python.org/issue8288 georg.brandl I HAVE A PROBLEM WITH PYTHON 0 days http://bugs.python.org/issue8290 ezio.melotti i have a doubt with using __init__ and .self and classes 0 days http://bugs.python.org/issue8291 ezio.melotti Allow Fraction constructor to accept float and decimal instanc 1 days http://bugs.python.org/issue8294 mark.dickinson patch in what way we have to save tha module? 0 days http://bugs.python.org/issue8298 eric.smith Allow struct.pack to handle objects with an __index__ method. 2 days http://bugs.python.org/issue8300 mark.dickinson patch test_pep263 failure on OS X 1 days http://bugs.python.org/issue8307 mark.dickinson raw_bytes.decode('cp932') -- spurious mappings 1 days http://bugs.python.org/issue8308 r.david.murray Sin(x) is Wrong 1 days http://bugs.python.org/issue8309 derekroconnor dis.dis function skips new-style classes in a module 0 days http://bugs.python.org/issue8310 benjamin.peterson test_ctypes fails in test_ulonglong on sparc buildbots 3 days http://bugs.python.org/issue8314 theller patch, buildbot test_gdb is susceptible to tty width settings 0 days http://bugs.python.org/issue8316 r.david.murray buildbot test_tarfile fails intermittently on Windows 0 days http://bugs.python.org/issue8317 brian.curtin docs on socket.recv_into doesn't mention the return value 1 days http://bugs.python.org/issue8320 georg.brandl easy Give access to openssl version number 0 days http://bugs.python.org/issue8321 giampaolo.rodola patch unintuitive behaviour of logging message propagation 1 days http://bugs.python.org/issue8327 pakal longobject.c: silence warnings (Visual Studio) 1 days http://bugs.python.org/issue8328 skrah patch select.select() can return lists with identical id()'s 0 days http://bugs.python.org/issue8329 benjamin.peterson a documentation grammar fix in logging module 0 days http://bugs.python.org/issue8331 vinay.sajip patch test_multiprocessing: pickling failures 0 days http://bugs.python.org/issue8333 michael.foord patch, buildbot winreg.QueryValue should return bytes, not unicode 0 days http://bugs.python.org/issue8334 stutzbach PyObject_CallObject - Not "reference-count-neutral" 1 days http://bugs.python.org/issue8336 georg.brandl test_gdb fails on Sparc Ubuntu 0 days http://bugs.python.org/issue8337 dmalcolm buildbot sphinx bug? 0 days http://bugs.python.org/issue8341 georg.brandl Python fails to run if launched from NTFS symlink and DLL not 0 days http://bugs.python.org/issue8342 jaraco patch missing method MatchObject.groups in re module doc 0 days http://bugs.python.org/issue8345 brian.curtin string capitalize erroneously lower-case any non-first letters 0 days http://bugs.python.org/issue8347 pjenvey test_urllib2net fails because of bad URL 0 days http://bugs.python.org/issue8348 loewis buildbot os.environ.get returns incorrect data 0 days http://bugs.python.org/issue8349 loewis imp.find_module of a .py ending dir causes glibc double free c 0 days http://bugs.python.org/issue8352 flox Negative exponentiation behaving oddly in python shell 0 days http://bugs.python.org/issue8353 CWardUSC SyntaxError: integer assignment with leading zeros (only 8 and 0 days http://bugs.python.org/issue8356 merwok Conversion of longs to bytes and vice-versa. 82 days http://bugs.python.org/issue1023290 mark.dickinson patch Compile of _socket fails on IRIX with 2.4 1933 days http://bugs.python.org/issue1086642 georg.brandl Add write buffering to gzip 1398 days http://bugs.python.org/issue1501108 pitrou PEP101/102 out of date 1366 days http://bugs.python.org/issue1518617 georg.brandl struct.pack raises TypeError where it used to convert 33 days http://bugs.python.org/issue1530559 mark.dickinson patch Top Issues Most Discussed (10) ______________________________ 21 test_ftplib fails with OpenSSL 0.9.8m 30 days open http://bugs.python.org/issue8108 15 Improve GIL in 2.7 6 days open http://bugs.python.org/issue8299 15 SocketServer doesn't handle syscall interruption 47 days open http://bugs.python.org/issue7978 11 test_multiprocessing: pickling failures 0 days closed http://bugs.python.org/issue8333 11 Give access to openssl version number 0 days closed http://bugs.python.org/issue8321 11 TestLoader.loadTestsFromName swallows import errors 109 days open http://bugs.python.org/issue7559 10 test_pep263 failure on OS X 1 days closed http://bugs.python.org/issue8307 10 Allow struct.pack to handle objects with an __index__ method. 2 days closed http://bugs.python.org/issue8300 9 Add gdb7 hooks to make it easier to debug Python 34 days closed http://bugs.python.org/issue8032 9 Add os.symlink() and os.path.islink() support for Windows 1271 days open http://bugs.python.org/issue1578269 From barry at python.org Fri Apr 9 22:24:10 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 9 Apr 2010 16:24:10 -0400 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() Message-ID: <20100409162410.29ed71f2@heresy> I've run into a minor snag implementing the __cached__ attribute on imported modules. From PEP 3147: As part of this PEP, we will add an `__cached__` attribute to modules, which will always point to the actual `pyc` file that was read or written. When the environment variable `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if the source lives on a read-only filesystem, then the `__cached__` attribute will point to the location that the `pyc` file *would* have been written to if it didn't exist. This location of course includes the `__pycache__` subdirectory in its path. The right place to add this seems to be PyImport_ExecCodeModuleEx(), which passes in a `pathname` argument. This function is not documented in the C API reference manual and about the only place where it's described is Misc/HISTORY: - New function PyImport_ExecCodeModuleEx(), which extends PyImport_ExecCodeModule() by adding an extra parameter to pass it the true file. The "true file" can either be the source .py file, the legacy .pyc file, or the PEP 3147 .pyc file depending on the circumstances. The caller knows which it is, but that function itself doesn't. I've tentatively worked out some code that lets it guess, but it's a kludge, it's ugly and I don't like it. I think the right fix is to extend PyImport_ExecCodeModuleEx() to also pass in the pathname for __cached__ (or NULL if None is appropriate). It bothers me a little to change this API, but OTOH, it's an *undocumented* API, so I don't feel too badly. ;) Since this is one of the last things to implement for PEP 3147, I thought I'd ask and see if anybody had any better suggestions. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Fri Apr 9 23:52:16 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Apr 2010 14:52:16 -0700 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: <20100409162410.29ed71f2@heresy> References: <20100409162410.29ed71f2@heresy> Message-ID: It may be undocumented but it doesn't start with _ and it exists to preserve backwards compatibility. So I recommend adding PyImport_ExecCodeModuleExEx(). On Fri, Apr 9, 2010 at 1:24 PM, Barry Warsaw wrote: > I've run into a minor snag implementing the __cached__ attribute on imported > modules. ?From PEP 3147: > > ? ?As part of this PEP, we will add an `__cached__` attribute to modules, > ? ?which will always point to the actual `pyc` file that was read or > ? ?written. ?When the environment variable `$PYTHONDONTWRITEBYTECODE` is > ? ?set, or the `-B` option is given, or if the source lives on a > ? ?read-only filesystem, then the `__cached__` attribute will point to > ? ?the location that the `pyc` file *would* have been written to if it > ? ?didn't exist. ?This location of course includes the `__pycache__` > ? ?subdirectory in its path. > > The right place to add this seems to be PyImport_ExecCodeModuleEx(), which > passes in a `pathname` argument. ?This function is not documented in the C API > reference manual and about the only place where it's described is > Misc/HISTORY: > > ? ?- New function PyImport_ExecCodeModuleEx(), which extends > ? ?PyImport_ExecCodeModule() by adding an extra parameter to pass it the > ? ?true file. > > The "true file" can either be the source .py file, the legacy .pyc file, or > the PEP 3147 .pyc file depending on the circumstances. ?The caller knows which > it is, but that function itself doesn't. ?I've tentatively worked out some > code that lets it guess, but it's a kludge, it's ugly and I don't like it. ?I > think the right fix is to extend PyImport_ExecCodeModuleEx() to also pass in > the pathname for __cached__ (or NULL if None is appropriate). > > It bothers me a little to change this API, but OTOH, it's an *undocumented* > API, so I don't feel too badly. ;) Since this is one of the last things to > implement for PEP 3147, I thought I'd ask and see if anybody had any better > suggestions. > > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From barry at python.org Sat Apr 10 00:00:50 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 9 Apr 2010 18:00:50 -0400 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: References: <20100409162410.29ed71f2@heresy> Message-ID: <20100409180050.122d6d42@heresy> On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: >It may be undocumented but it doesn't start with _ and it exists to >preserve backwards compatibility. So I recommend adding >PyImport_ExecCodeModuleExEx(). Cool, thanks. Now I can't wait for PyImport_ExecCodeModuleExExEx() :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Sat Apr 10 00:40:26 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 9 Apr 2010 18:40:26 -0400 Subject: [Python-Dev] PEP 3147 working implementation In-Reply-To: <20100408090815.2c7caf40@heresy> References: <20100401161242.6405053d@heresy> <20100406214031.5f5143ec@heresy> <20100408090815.2c7caf40@heresy> Message-ID: <20100409184026.18996e2f@heresy> Patch set 4 has been uploaded to Rietveld: http://codereview.appspot.com/842043/show This includes some fixes for Windows and support for the __cached__ attribute on modules. While I need to do another pass through the PEP to make sure I've gotten everything, this code is very nearly feature complete, so it's probably worth getting Guido to pronounce on the PEP pretty soon. -Barry P.S. 'bzr branch lp:~barry/python/pep3147' -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From p.f.moore at gmail.com Sat Apr 10 00:54:52 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Apr 2010 23:54:52 +0100 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: <20100409180050.122d6d42@heresy> References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> Message-ID: On 9 April 2010 23:00, Barry Warsaw wrote: > On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: > >>It may be undocumented but it doesn't start with _ and it exists to >>preserve backwards compatibility. So I recommend adding >>PyImport_ExecCodeModuleExEx(). > > Cool, thanks. ?Now I can't wait for PyImport_ExecCodeModuleExExEx() :) Would it be better to name this one _PyImport_ExecCodeModuleExEx (with an underscore) so that we *don't* need to create an ExExEx version in future? (Sorry, Barry :-)) Paul. From solipsis at pitrou.net Sat Apr 10 01:05:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Apr 2010 23:05:28 +0000 (UTC) Subject: [Python-Dev] PEP 3147 References: <20100409162410.29ed71f2@heresy> Message-ID: Hello, I'm sorry to chime in a bit late about an aspect of the PEP, but I only wondered about it while reviewing your code :-) ? Instead, this PEP proposes to add a mapping between internal magic numbers and a user-friendly tag. Newer versions of Python can add to this mapping so that all later Pythons know the mapping between tags and magic numbers. ? The question is: why do we have to keep a mapping of past tags and magic numbers? Don't we only care about our current tag and magic number? (similarly, we don't know, and need to know, about Jython's or Pypy's stuff...). As far as I can tell, it would remove the burden of maintening an ever-growing registry of past magic numbers and tags. Regards Antoine. From guido at python.org Sat Apr 10 02:41:48 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Apr 2010 17:41:48 -0700 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> Message-ID: On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: > On 9 April 2010 23:00, Barry Warsaw wrote: >> On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: >> >>>It may be undocumented but it doesn't start with _ and it exists to >>>preserve backwards compatibility. So I recommend adding >>>PyImport_ExecCodeModuleExEx(). >> >> Cool, thanks. ?Now I can't wait for PyImport_ExecCodeModuleExExEx() :) > > Would it be better to name this one _PyImport_ExecCodeModuleExEx (with > an underscore) so that we *don't* need to create an ExExEx version in > future? (Sorry, Barry :-)) I don't care about what name you pick, and my ExEx proposal was meant to include half a wink, but http://docs.python.org/c-api/import.html makes it clear that PyImport_ExecCodeModuleEx() is far from private! (I don't know where Barry got that idea.) While Google Code Search finds mostly references to PyImport_ExecCodeModuleEx in the Python source code and various copies of it, it also shows some real uses, e.g. http://www.google.com/codesearch/p?hl=en#bkFK9YpaWlI/ubuntu/pool/universe/y/yehia/yehia_0.5.4.orig.tar.gz|PZ0_Xf7QzC0/yehia-0.5.4.orig/plugins/python/python-loader.cc&q=PyImport_ExecCodeModuleEx -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Apr 10 17:51:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Apr 2010 01:51:19 +1000 Subject: [Python-Dev] Status of 2.7b1? Message-ID: <4BC09E77.8030209@gmail.com> The trunk's been frozen for a few days now, which is starting to cut into the window for new fixes between b1 and b2 (down to just under 3 weeks, and that's only if b1 was ready for release today). Is work in train to address or document the remaining buildbot failures (e.g. test_os on Windows [1]). At what point do we decide to document something as a known defect in the beta and release it anyway? (My question is mostly aimed at Benjamin for obvious reasons, but it would be good to hear from anyone that is actually looking into the Windows buildbot failure) Cheers, Nick. [1] http://www.python.org/dev/buildbot/trunk/builders/x86%20XP-4%20trunk/builds/3380/steps/test/logs/stdio -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Sat Apr 10 17:51:22 2010 From: barry at python.org (Barry Warsaw) Date: Sat, 10 Apr 2010 11:51:22 -0400 Subject: [Python-Dev] PEP 3147 In-Reply-To: References: <20100409162410.29ed71f2@heresy> Message-ID: <20100410115122.7f6ad18d@heresy> On Apr 09, 2010, at 11:05 PM, Antoine Pitrou wrote: >? Instead, this PEP proposes to add a mapping between internal magic numbers >and a user-friendly tag. Newer versions of Python can add to this mapping so >that all later Pythons know the mapping between tags and magic numbers. ? > >The question is: why do we have to keep a mapping of past tags and magic >numbers? Don't we only care about our current tag and magic number? >(similarly, we don't know, and need to know, about Jython's or Pypy's >stuff...). > >As far as I can tell, it would remove the burden of maintening an ever-growing >registry of past magic numbers and tags. If you look at the comment near the top of import.c, we kind of do anyway, we just don't make it available to Python. ;) I don't have strong feelings about this. I thought it would be handy for future Python's to have access to this, but then, without access to previous version magic numbers, it probably doesn't help much. And as you say, CPython won't know about alternative implementation's tags. So I'm willing to call YAGNI on it and just expose the current Python's magic tag. While we're at it, how about making both the tag and the number attributes of the imp module, instead of functions like .get_magic()? Of course we'd keep the latter for backward compatibility. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Sat Apr 10 17:58:41 2010 From: barry at python.org (Barry Warsaw) Date: Sat, 10 Apr 2010 11:58:41 -0400 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> Message-ID: <20100410115841.5732ab97@heresy> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >> an underscore) so that we *don't* need to create an ExExEx version in >> future? (Sorry, Barry :-)) > >I don't care about what name you pick, and my ExEx proposal was meant >to include half a wink, but http://docs.python.org/c-api/import.html >makes it clear that PyImport_ExecCodeModuleEx() is far from private! >(I don't know where Barry got that idea.) Note that it's the non-Ex version that's documented here. AFAICT, PyImport_ExecCodeModuleEx() is not documented. I'm happy to fix that in my branch as well. >While Google Code Search >finds mostly references to PyImport_ExecCodeModuleEx in the Python >source code and various copies of it, it also shows some real uses, >e.g. >http://www.google.com/codesearch/p?hl=en#bkFK9YpaWlI/ubuntu/pool/universe/y/yehia/yehia_0.5.4.orig.tar.gz|PZ0_Xf7QzC0/yehia-0.5.4.orig/plugins/python/python-loader.cc&q=PyImport_ExecCodeModuleEx Sure, let's not break existing API even if it's undocumented. The one nice thing about ExEx() is that it's clearly related to the two previous API functions its based on. But if you don't like it then how about something like PyImport_ExecCodeModuleWithPathnames()? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brian.curtin at gmail.com Sat Apr 10 18:02:39 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sat, 10 Apr 2010 11:02:39 -0500 Subject: [Python-Dev] Status of 2.7b1? In-Reply-To: <4BC09E77.8030209@gmail.com> References: <4BC09E77.8030209@gmail.com> Message-ID: On Sat, Apr 10, 2010 at 10:51, Nick Coghlan wrote: > The trunk's been frozen for a few days now, which is starting to cut > into the window for new fixes between b1 and b2 (down to just under 3 > weeks, and that's only if b1 was ready for release today). > > Is work in train to address or document the remaining buildbot failures > (e.g. test_os on Windows [1]). At what point do we decide to document > something as a known defect in the beta and release it anyway? > > (My question is mostly aimed at Benjamin for obvious reasons, but it > would be good to hear from anyone that is actually looking into the > Windows buildbot failure) > > Cheers, > Nick. > > [1] > > http://www.python.org/dev/buildbot/trunk/builders/x86%20XP-4%20trunk/builds/3380/steps/test/logs/stdio > > I contacted David Bolen for some details about the his buildbot because I can't reproduce the failure on any Windows XP, Server 2003, or 7 box that I have, and it's also not a problem on the other XP buildbot. He's traveling at the moment but will try to get me access to the box after the weekend is over. When manually running test_os how buildbot runs it (via test.bat, which runs rt.bat) he sees the failure. When running the test on a clean checkout outside of how buildbot does anything, he doesn't see the failure. I'll try to get access to figure out what the difference is. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Apr 10 18:04:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Apr 2010 16:04:24 +0000 (UTC) Subject: [Python-Dev] Status of 2.7b1? References: <4BC09E77.8030209@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > Is work in train to address or document the remaining buildbot failures > (e.g. test_os on Windows [1]). At what point do we decide to document > something as a known defect in the beta and release it anyway? I'm not handling the test_os issue (which I think is in Brian Curtin's hands), but as far as test_ftplib is concerned (behaviour change with newest OpenSSL versions), it was decided to address the issue after the beta. Regards Antoine. From guido at python.org Sat Apr 10 18:12:38 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Apr 2010 09:12:38 -0700 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: <20100410115841.5732ab97@heresy> References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> <20100410115841.5732ab97@heresy> Message-ID: On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: > On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: > >>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >>> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >>> an underscore) so that we *don't* need to create an ExExEx version in >>> future? (Sorry, Barry :-)) >> >>I don't care about what name you pick, and my ExEx proposal was meant >>to include half a wink, but http://docs.python.org/c-api/import.html >>makes it clear that PyImport_ExecCodeModuleEx() is far from private! >>(I don't know where Barry got that idea.) > > Note that it's the non-Ex version that's documented here. ?AFAICT, > PyImport_ExecCodeModuleEx() is not documented. ?I'm happy to fix that in my > branch as well. Ah, true. And yes, please. >>While Google Code Search >>finds mostly references to PyImport_ExecCodeModuleEx in the Python >>source code and various copies of it, it also shows some real uses, >>e.g. >>http://www.google.com/codesearch/p?hl=en#bkFK9YpaWlI/ubuntu/pool/universe/y/yehia/yehia_0.5.4.orig.tar.gz|PZ0_Xf7QzC0/yehia-0.5.4.orig/plugins/python/python-loader.cc&q=PyImport_ExecCodeModuleEx > > Sure, let's not break existing API even if it's undocumented. ?The one nice > thing about ExEx() is that it's clearly related to the two previous API > functions its based on. ?But if you don't like it then how about something > like PyImport_ExecCodeModuleWithPathnames()? Sure. -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Sat Apr 10 18:20:13 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 10 Apr 2010 11:20:13 -0500 Subject: [Python-Dev] Status of 2.7b1? In-Reply-To: <4BC09E77.8030209@gmail.com> References: <4BC09E77.8030209@gmail.com> Message-ID: 2010/4/10 Nick Coghlan : > The trunk's been frozen for a few days now, which is starting to cut > into the window for new fixes between b1 and b2 (down to just under 3 > weeks, and that's only if b1 was ready for release today). > > Is work in train to address or document the remaining buildbot failures > (e.g. test_os on Windows [1]). At what point do we decide to document > something as a known defect in the beta and release it anyway? I'm going to do it now (today). We've deffered basically all the failures until after the beta, so I think I'll just release now. -- Regards, Benjamin From ncoghlan at gmail.com Sat Apr 10 18:39:00 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Apr 2010 02:39:00 +1000 Subject: [Python-Dev] PEP 3147 In-Reply-To: <20100410115122.7f6ad18d@heresy> References: <20100409162410.29ed71f2@heresy> <20100410115122.7f6ad18d@heresy> Message-ID: <4BC0A9A4.2040307@gmail.com> Barry Warsaw wrote: > I don't have strong feelings about this. I thought it would be handy for > future Python's to have access to this, but then, without access to previous > version magic numbers, it probably doesn't help much. And as you say, CPython > won't know about alternative implementation's tags. > > So I'm willing to call YAGNI on it and just expose the current Python's magic > tag. While we're at it, how about making both the tag and the number > attributes of the imp module, instead of functions like .get_magic()? Of > course we'd keep the latter for backward compatibility. I think one of the virtues of the functions is making it bleedingly obvious to all concerned that these are read only values. So +1 to only exposing the current version of the implementation tag and magic number, and +0 to doing so via attributes rather than functions. (I'm still in favour of keeping the list of old tags and magic numbers in a source comment though - commenting them out rather than deleting them when updating them isn't a major hassle). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rnk at mit.edu Sat Apr 10 20:06:59 2010 From: rnk at mit.edu (Reid Kleckner) Date: Sat, 10 Apr 2010 14:06:59 -0400 Subject: [Python-Dev] Tuning Python dicts Message-ID: Hey folks, I was looking at tuning Python dicts for a data structures class final project. I've looked through Object/dictnotes.txt, and obviously there's already a large body of work on this topic. My idea was to alter dict collision resolution as described in the hopscotch hashing paper[1]. I think the PDF I have came from behind a pay-wall, so I can't find a link to the original paper. [1] http://en.wikipedia.org/wiki/Hopscotch_hashing Just to be clear, this is an experiment I'm doing for a class. If it is successful, which I think is unlikely since Python dicts are already well-tuned, I might consider trying to contribute it back to CPython over the summer. The basic idea of hopscotch hashing is to use linear probing with a cutoff (H), but instead of rehashing when the probe fails, find the next empty space in the table and move it into the neighborhood of the original hash index. This means you have to spend potentially a lot of extra time during insertion, but it makes lookups very fast because H is usually chosen such that the entire probe spans at most two cache lines. This is much better than the current random (what's the right name for the current approach?) probing solution, which does potentially a handful of random accesses into the table. Looking at dictnotes.txt, I can see that people have experimented with taking advantage of cache locality. I was wondering what benchmarks were used to glean these lessons before I write my own. Python obviously has very particular workloads that need to be modeled appropriately, such as namespaces and **kwargs dicts. Any other advice would also be helpful. Thanks, Reid One caveat I need to work out: If more than H items collide into a single bucket, then you need to rehash. However, if you have a particularly evil hash function which always returns zero, no matter how much you rehash, you will never be able to fit all the items into the first H buckets. This would cause an infinite loop, while I believe the current solution will simply have terrible performance. IMO the solution is just to increase H for the table if the rehash fails, but realistically, this will never happen unless the programmer is being evil. I'd probably skip this detail for the class implementation. From g.brandl at gmx.net Sat Apr 10 20:28:12 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 10 Apr 2010 20:28:12 +0200 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> <20100410115841.5732ab97@heresy> Message-ID: Am 10.04.2010 18:12, schrieb Guido van Rossum: > On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: >> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >> >>>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >>>> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >>>> an underscore) so that we *don't* need to create an ExExEx version in >>>> future? (Sorry, Barry :-)) >>> >>>I don't care about what name you pick, and my ExEx proposal was meant >>>to include half a wink, but http://docs.python.org/c-api/import.html >>>makes it clear that PyImport_ExecCodeModuleEx() is far from private! >>>(I don't know where Barry got that idea.) >> >> Note that it's the non-Ex version that's documented here. AFAICT, >> PyImport_ExecCodeModuleEx() is not documented. I'm happy to fix that in my >> branch as well. > > Ah, true. And yes, please. *cough* http://docs.python.org/dev/c-api/import.html#PyImport_ExecCodeModuleEx Not backported to stable yet, though. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From tjgolden at gmail.com Sat Apr 10 20:37:14 2010 From: tjgolden at gmail.com (Tim Golden) Date: Sat, 10 Apr 2010 19:37:14 +0100 Subject: [Python-Dev] Status of 2.7b1? In-Reply-To: References: <4BC09E77.8030209@gmail.com> Message-ID: <4BC0C55A.5060001@gmail.com> On 10/04/2010 17:02, Brian Curtin wrote: > I contacted David Bolen for some details about the his buildbot because I > can't reproduce the failure on any Windows XP, Server 2003, or 7 box that I > have, and it's also not a problem on the other XP buildbot. He's traveling > at the moment but will try to get me access to the box after the weekend is > over. Might it be significant that he's running a Cygwin build of Python? I've only run the tests on a native Win32 build myself, and I imagine that's true for you too... TJG From brian.curtin at gmail.com Sat Apr 10 20:47:09 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sat, 10 Apr 2010 13:47:09 -0500 Subject: [Python-Dev] Status of 2.7b1? In-Reply-To: <4BC0C55A.5060001@gmail.com> References: <4BC09E77.8030209@gmail.com> <4BC0C55A.5060001@gmail.com> Message-ID: On Sat, Apr 10, 2010 at 13:37, Tim Golden wrote: > On 10/04/2010 17:02, Brian Curtin wrote: > >> I contacted David Bolen for some details about the his buildbot because I >> can't reproduce the failure on any Windows XP, Server 2003, or 7 box that >> I >> have, and it's also not a problem on the other XP buildbot. He's traveling >> at the moment but will try to get me access to the box after the weekend >> is >> over. >> > > Might it be significant that he's running a Cygwin build of Python? > I've only run the tests on a native Win32 build myself, and I imagine > that's true for you too... > > > TJG > > The tests are run on a native Win32 build as compiled by VS2008. The functionality is Win32 specific and wouldn't work on Cygwin, so the tests are skipped there. I believe Cygwin is used for kicking off the tests and other buildbot stuff, but they don't actually run through Cygwin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Apr 10 20:52:16 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 10 Apr 2010 13:52:16 -0500 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 Message-ID: On behalf of the Python development team, I'm merry to announce the first beta release of Python 2.7. Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version in the 2.x series. Though more major releases have not been absolutely ruled out, it's likely that the 2.7 release will an extended period of maintenance for the 2.x series. 2.7 includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, set literals, dictionary views, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, a new sysconfig module, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ While this is a development release and is thus not suitable for production use, we encourage Python application and library developers to test the release with their code and report any bugs they encounter. The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Enjoy! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From deniskolodin at gmail.com Sat Apr 10 20:53:38 2010 From: deniskolodin at gmail.com (Denis Kolodin) Date: Sat, 10 Apr 2010 22:53:38 +0400 Subject: [Python-Dev] Python-Dev Digest, Vol 81, Issue 31 In-Reply-To: References: Message-ID: Hello! My name is Denis Kolodin. I live in Russia, Tambov. I was developing much time with C, Java, C#, R. But two month ago I'm using Python. It's really cool. Now, I move ALL my projects to it fully and have some ideas which API's extensions may will be useful. The first thing I want to say about is an extension of CSV api. In R language I could to set types for the every column in a csv file. I propose to add a same function to the Python's standard library. Here it is (Python 3 version): import csv def reader2(csvfile, frame, *delimiter**=**';'*, **fmtparams): reader = csv.reader(csvfile, delimiter=delimiter, **fmtparams) for row in reader: l = min(len(row), len(frame)) yield [frame[idx](row[idx]) for idx in range(l)] This's generator function which converts an every column to the associated type. In *frame *argument you must to set tuple/list of functions which will uses to convert values in same positions of row from csv file. Frame looks like list of types ))) By default it uses ';' delimiter to make float values conversion are possible. As a sample you have the csv file like: *Any spam...; 1; 2.0; 3* I've saved it to "sample.csv" :) If you are using function reader in the standard "csv" module you get rows as a list of strings :( *>>> reader = csv.reader(open("sample.csv"), delimiter=";")* *>>> print(next(reader))* *['Any spam...', ' 1', ' 2.0', ' 3']* * * * It's not bad in certan situatiuons. But with "reader2" function you can get a list with necessary types: >>> reader = reader2(open("foodstuffs.csv"), (str, int, float, int)) >>> print(next(reader)) ['Any spam...', 1, 2.0, 3] Now you can work with items without extra conversions. [?] I think it's good to add this function to the standard library. I've already used it many times. This function can be useful for many people who works with csv files. And I suppose it conforms to "batteries included" philosophy. What do you think about this extension? Is it possible to add this function to standard library or to add the same behavior to the standard "readed" function in "csv" Python's module? Best Regards, Denis Kolodin Russia, Tambov * 2010/4/10 > Send Python-Dev mailing list submissions to > python-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-dev > or, via email, send a message with subject or body 'help' to > python-dev-request at python.org > > You can reach the person managing the list at > python-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > > Today's Topics: > > 1. Re: PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() > (Guido van Rossum) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 9 Apr 2010 17:41:48 -0700 > From: Guido van Rossum > To: Paul Moore > Cc: Python-Dev Dev > Subject: Re: [Python-Dev] PEP 3147, __cached__, and > PyImport_ExecCodeModuleEx() > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: > > On 9 April 2010 23:00, Barry Warsaw wrote: > >> On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: > >> > >>>It may be undocumented but it doesn't start with _ and it exists to > >>>preserve backwards compatibility. So I recommend adding > >>>PyImport_ExecCodeModuleExEx(). > >> > >> Cool, thanks. ?Now I can't wait for PyImport_ExecCodeModuleExExEx() :) > > > > Would it be better to name this one _PyImport_ExecCodeModuleExEx (with > > an underscore) so that we *don't* need to create an ExExEx version in > > future? (Sorry, Barry :-)) > > I don't care about what name you pick, and my ExEx proposal was meant > to include half a wink, but http://docs.python.org/c-api/import.html > makes it clear that PyImport_ExecCodeModuleEx() is far from private! > (I don't know where Barry got that idea.) While Google Code Search > finds mostly references to PyImport_ExecCodeModuleEx in the Python > source code and various copies of it, it also shows some real uses, > e.g. > > http://www.google.com/codesearch/p?hl=en#bkFK9YpaWlI/ubuntu/pool/universe/y/yehia/yehia_0.5.4.orig.tar.gz|PZ0_Xf7QzC0/yehia-0.5.4.orig/plugins/python/python-loader.cc&q=PyImport_ExecCodeModuleEx > > -- > --Guido van Rossum (python.org/~guido) > > > ------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > > > End of Python-Dev Digest, Vol 81, Issue 31 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 360.gif Type: image/gif Size: 453 bytes Desc: not available URL: From martin at v.loewis.de Sat Apr 10 20:57:58 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Apr 2010 20:57:58 +0200 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: References: Message-ID: <4BC0CA36.2030707@v.loewis.de> > Any other advice would also be helpful. I may be missing the point, but ISTM that the assumption of this approach is that there are often collisions in the hash table. I think that assumption is false; at least, I recommend to validate that assumption before proceeding. There are, of course, cases where dicts do show collisions (namely when all keys hash equal), however, I'm uncertain whether the approach would help in that case. Regards, Martin From solipsis at pitrou.net Sat Apr 10 20:57:54 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Apr 2010 18:57:54 +0000 (UTC) Subject: [Python-Dev] [RELEASED] 2.7 beta 1 References: Message-ID: Benjamin Peterson python.org> writes: > > On behalf of the Python development team, I'm merry to announce the first beta > release of Python 2.7. Congratulations, and thanks for your patience :) From rnk at mit.edu Sat Apr 10 22:32:25 2010 From: rnk at mit.edu (Reid Kleckner) Date: Sat, 10 Apr 2010 16:32:25 -0400 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: <4BC0CA36.2030707@v.loewis.de> References: <4BC0CA36.2030707@v.loewis.de> Message-ID: On Sat, Apr 10, 2010 at 2:57 PM, "Martin v. L?wis" wrote: >> Any other advice would also be helpful. > > I may be missing the point, but ISTM that the assumption of this > approach is that there are often collisions in the hash table. I think > that assumption is false; at least, I recommend to validate that > assumption before proceeding. It's just an experiment for a class, not something I am (yet) seriously thinking about contributing back to CPython. I think my chances of improving over the current implementation are slim. I do not have that much hubris. :) I would just rather do experimental rather than theoretical work with data structures. I think you're right about the number of collisions, though. CPython dicts use a pretty low load factor (2/3) to keep collision counts down. One of the major benefits cited in the paper is the ability to maintain performance in the face of higher load factors, so I may be able to bump up the load factor to save memory. This would increase collisions, but then that wouldn't matter, because resolving them would only require looking within two consecutive cache lines. > There are, of course, cases where dicts do show collisions (namely when > all keys hash equal), however, I'm uncertain whether the approach would > help in that case. Yes, in fact, hopscotch hashing fails completely as I mentioned at the end of my last message. Reid From solipsis at pitrou.net Sat Apr 10 22:40:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Apr 2010 20:40:26 +0000 (UTC) Subject: [Python-Dev] Tuning Python dicts References: <4BC0CA36.2030707@v.loewis.de> Message-ID: Reid Kleckner mit.edu> writes: > > I think you're right about the number of collisions, though. CPython > dicts use a pretty low load factor (2/3) to keep collision counts > down. One of the major benefits cited in the paper is the ability to > maintain performance in the face of higher load factors, so I may be > able to bump up the load factor to save memory. This would increase > collisions, but then that wouldn't matter, because resolving them > would only require looking within two consecutive cache lines. Why wouldn't it matter? Hash collisions still involve more CPU work, even though if you're not access memory a lot. Antoine. From solipsis at pitrou.net Sat Apr 10 22:45:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Apr 2010 20:45:35 +0000 (UTC) Subject: [Python-Dev] Tuning Python dicts References: <4BC0CA36.2030707@v.loewis.de> Message-ID: Antoine Pitrou pitrou.net> writes: > > Why wouldn't it matter? Hash collisions still involve more CPU work, even though > if you're not access memory a lot. (sorry for the awful grammar in the last sentence) From rnk at mit.edu Sat Apr 10 23:35:24 2010 From: rnk at mit.edu (Reid Kleckner) Date: Sat, 10 Apr 2010 17:35:24 -0400 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: References: <4BC0CA36.2030707@v.loewis.de> Message-ID: On Sat, Apr 10, 2010 at 4:40 PM, Antoine Pitrou wrote: > Reid Kleckner mit.edu> writes: >> >> I think you're right about the number of collisions, though. ?CPython >> dicts use a pretty low load factor (2/3) to keep collision counts >> down. ?One of the major benefits cited in the paper is the ability to >> maintain performance in the face of higher load factors, so I may be >> able to bump up the load factor to save memory. ?This would increase >> collisions, but then that wouldn't matter, because resolving them >> would only require looking within two consecutive cache lines. > > Why wouldn't it matter? Hash collisions still involve more CPU work, even though > if you're not access memory a lot. So we know for sure that extra CPU work to avoid cache misses is a big win, but it's never clear whether or not any given memory access will be a miss. Because Python's access patterns are rarely random, it may turn out that all of the elements it accesses are in cache and the extra CPU work dominates. If you have a random access pattern over a large dataset, the dictionary will not fit in cache, and saving random memory accesses at the expense of CPU will be a win. Reid From dreamingforward at gmail.com Sun Apr 11 00:13:49 2010 From: dreamingforward at gmail.com (average) Date: Sat, 10 Apr 2010 17:13:49 -0500 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 In-Reply-To: References: Message-ID: > On behalf of the Python development team, I'm merry to announce the first beta > release of Python 2.7. > > Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version > in the 2.x series. ?Though more major releases have not been absolutely ruled > out, it's likely that the 2.7 release will an extended period of maintenance for > the 2.x series. May I propose that the developers consider keeping this release *beta* until after the present Python moratorium? That is, don't let it be marked as *official* until after, say, Python 3.3. There are so many features taken from 3.0 that I fear that it will postpone its adoption interminably (it is, in practice, treated as "beta" software itself). By making it doctrine that it won't be official until the next "major" Python release, it will encourage those who are able, to just make the jump to 3.0, while those who cannot will have the subtle pressure to make the shift, however gradual. Additionally, it will give the community further incentive to make Python3 all that it was intended to be. Personally, the timing of v3 prevented me from fully participating in that effort, and, not ignoring the work of those who did contribute, I think many of us feel that it has not reached its potential. Just a small suggestion... .. . marcos From barry at python.org Sun Apr 11 00:25:53 2010 From: barry at python.org (Barry Warsaw) Date: Sat, 10 Apr 2010 18:25:53 -0400 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> <20100410115841.5732ab97@heresy> Message-ID: <20100410182553.669fefee@heresy> On Apr 10, 2010, at 08:28 PM, Georg Brandl wrote: >Am 10.04.2010 18:12, schrieb Guido van Rossum: >> On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: >>> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >>> >>>>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >>>>> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >>>>> an underscore) so that we *don't* need to create an ExExEx version in >>>>> future? (Sorry, Barry :-)) >>>> >>>>I don't care about what name you pick, and my ExEx proposal was meant >>>>to include half a wink, but http://docs.python.org/c-api/import.html >>>>makes it clear that PyImport_ExecCodeModuleEx() is far from private! >>>>(I don't know where Barry got that idea.) >>> >>> Note that it's the non-Ex version that's documented here. AFAICT, >>> PyImport_ExecCodeModuleEx() is not documented. I'm happy to fix that in my >>> branch as well. >> >> Ah, true. And yes, please. > >*cough* http://docs.python.org/dev/c-api/import.html#PyImport_ExecCodeModuleEx >Not backported to stable yet, though. 'dev' is Python 2.7 (i.e. trunk) right? I was looking at the py3k url, which it hasn't been ported to either yet I think. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From softw.devl at gmail.com Sun Apr 11 00:27:41 2010 From: softw.devl at gmail.com (Melton Low) Date: Sat, 10 Apr 2010 16:27:41 -0600 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 4:13 PM, average wrote: > > On behalf of the Python development team, I'm merry to announce the first > beta > > release of Python 2.7. > > > > Python 2.7 is scheduled (by Guido and Python-dev) to be the last major > version > > in the 2.x series. Though more major releases have not been absolutely > ruled > > out, it's likely that the 2.7 release will an extended period of > maintenance for > > the 2.x series. > > May I propose that the developers consider keeping this release *beta* > until after the present Python moratorium? That is, don't let it be > marked as *official* until after, say, Python 3.3. > > There are so many features taken from 3.0 that I fear that it will > postpone its adoption interminably (it is, in practice, treated as > "beta" software itself). By making it doctrine that it won't be > official until the next "major" Python release, it will encourage > those who are able, to just make the jump to 3.0, while those who > cannot will have the subtle pressure to make the shift, however > gradual. Additionally, it will give the community further incentive > to make Python3 all that it was intended to be. Personally, the > timing of v3 prevented me from fully participating in that effort, > and, not ignoring the work of those who did contribute, I think many > of us feel that it has not reached its potential. > > Just a small suggestion... .. . > > marcos > -- > http://mail.python.org/mailman/listinfo/python-list > I disagree. 2.7 should go GA as soon as the developers deemed it stable. Those who don't need 3rd party packages will no doubt migrate to 3.x. Those that required 3rd party packages not yet ported to 3.x will want to use 2.7. Delaying 2.7 from GA doesn't change the reality. I myself would want to use back ported features from 2.7 as a way to prepare for migration as soon as those 3rd party packages are ported to 3.x. Mel -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Apr 11 03:19:43 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Apr 2010 21:19:43 -0400 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: References: <4BC0CA36.2030707@v.loewis.de> Message-ID: >> I may be missing the point, but ISTM that the assumption of this >> approach is that there are often collisions in the hash table. I think >> that assumption is false; at least, I recommend to validate that >> assumption before proceeding. > > It's just an experiment for a class, not something I am (yet) > seriously thinking about contributing back to CPython. I think my > chances of improving over the current implementation are slim. I do > not have that much hubris. :) I would just rather do experimental > rather than theoretical work with data structures. > > I think you're right about the number of collisions, though. CPython > dicts use a pretty low load factor (2/3) to keep collision counts > down. There was a paper discussing Python dicts at PyCon 2010. I believe it included some data on collisions. I posted the link on Python list a couple of weeks ago. Terry Jan Reedy From tjreedy at udel.edu Sun Apr 11 03:23:21 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Apr 2010 21:23:21 -0400 Subject: [Python-Dev] Python-Dev Digest, Vol 81, Issue 31 In-Reply-To: References: Message-ID: On 4/10/2010 2:53 PM, Denis Kolodin wrote: > The first thing I want to say about is an extension of CSV api. I believe speculative proposals like this fit better on the python-list or python-ideas list. tjr From g.brandl at gmx.net Sun Apr 11 09:01:03 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 11 Apr 2010 09:01:03 +0200 Subject: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx() In-Reply-To: <20100410182553.669fefee@heresy> References: <20100409162410.29ed71f2@heresy> <20100409180050.122d6d42@heresy> <20100410115841.5732ab97@heresy> <20100410182553.669fefee@heresy> Message-ID: Am 11.04.2010 00:25, schrieb Barry Warsaw: > On Apr 10, 2010, at 08:28 PM, Georg Brandl wrote: > >>Am 10.04.2010 18:12, schrieb Guido van Rossum: >>> On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: >>>> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >>>> >>>>>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >>>>>> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >>>>>> an underscore) so that we *don't* need to create an ExExEx version in >>>>>> future? (Sorry, Barry :-)) >>>>> >>>>>I don't care about what name you pick, and my ExEx proposal was meant >>>>>to include half a wink, but http://docs.python.org/c-api/import.html >>>>>makes it clear that PyImport_ExecCodeModuleEx() is far from private! >>>>>(I don't know where Barry got that idea.) >>>> >>>> Note that it's the non-Ex version that's documented here. AFAICT, >>>> PyImport_ExecCodeModuleEx() is not documented. I'm happy to fix that in my >>>> branch as well. >>> >>> Ah, true. And yes, please. >> >>*cough* http://docs.python.org/dev/c-api/import.html#PyImport_ExecCodeModuleEx >>Not backported to stable yet, though. > > 'dev' is Python 2.7 (i.e. trunk) right? I was looking at the py3k url, which > it hasn't been ported to either yet I think. Yes. It was added in response to a ticket very recently, and therefore is only on trunk yet. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From deniskolodin at gmail.com Sun Apr 11 12:40:04 2010 From: deniskolodin at gmail.com (Denis Kolodin) Date: Sun, 11 Apr 2010 14:40:04 +0400 Subject: [Python-Dev] Python-Dev Digest, Vol 81, Issue 31 Message-ID: On Sun Apr 11 03:23:21 CEST 2010 Terry Reedy wrote: >?I believe speculative proposals like this fit better on the python-list or python-ideas list. I see. Thank you :) Denis Kolodin From regebro at gmail.com Sun Apr 11 21:23:32 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 11 Apr 2010 21:23:32 +0200 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 20:52, Benjamin Peterson wrote: > On behalf of the Python development team, I'm merry > to announce the first beta release of Python 2.7. Cool! -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From guido at python.org Sun Apr 11 21:36:54 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Apr 2010 12:36:54 -0700 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 11:57 AM, Antoine Pitrou wrote: > Benjamin Peterson python.org> writes: >> >> On behalf of the Python development team, I'm merry to announce the first beta >> release of Python 2.7. > > Congratulations, and thanks for your patience :) Congratulations indeed! Let me use this occasion to point out what an awesome community python-dev is. Not only do we have an great release manager (kudos to Benjamin for taking on this thankless job of herding cats), but the whole developer community has been contributing to some excellent releases. This makes me very happy and confident that Python will continue to thrive for many, many years, with or without my direct involvement in the details. While rumors of my retirement are greatly exaggerated, I am very glad to be able to leave the details to the community. Keep them coming, folks! -- --Guido van Rossum (python.org/~guido) From barry at python.org Sun Apr 11 19:36:19 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 11 Apr 2010 13:36:19 -0400 Subject: [Python-Dev] PEP 3147 In-Reply-To: <4BC0A9A4.2040307@gmail.com> References: <20100409162410.29ed71f2@heresy> <20100410115122.7f6ad18d@heresy> <4BC0A9A4.2040307@gmail.com> Message-ID: <20100411133619.5de1082f@heresy> On Apr 11, 2010, at 02:39 AM, Nick Coghlan wrote: >Barry Warsaw wrote: >> I don't have strong feelings about this. I thought it would be handy for >> future Python's to have access to this, but then, without access to previous >> version magic numbers, it probably doesn't help much. And as you say, CPython >> won't know about alternative implementation's tags. >> >> So I'm willing to call YAGNI on it and just expose the current Python's magic >> tag. While we're at it, how about making both the tag and the number >> attributes of the imp module, instead of functions like .get_magic()? Of >> course we'd keep the latter for backward compatibility. > >I think one of the virtues of the functions is making it bleedingly >obvious to all concerned that these are read only values. > >So +1 to only exposing the current version of the implementation tag and >magic number, and +0 to doing so via attributes rather than functions. > >(I'm still in favour of keeping the list of old tags and magic numbers >in a source comment though - commenting them out rather than deleting >them when updating them isn't a major hassle). Thinking about this more, I've decided to add imp.get_tag() and remove imp.magic_tags. This avoids changes to the existing API, and just adds to it in a more consistent way. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From nir at winpdb.org Mon Apr 12 00:50:14 2010 From: nir at winpdb.org (Nir Aides) Date: Mon, 12 Apr 2010 01:50:14 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: <20100313164651.7eb322b1@msiwind> References: <20100313164651.7eb322b1@msiwind> Message-ID: Hello all, I would like to kick this discussion back to life with a simplified implementation of the BFS scheduler, designed by the Linux kernel hacker Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt I submitted bfs.patch at http://bugs.python.org/issue7946. It is work in progress but is ready for some opinion. On my machine BFS gives comparable performance to gilinter, and seems to schedule threads more fairly, predictably, and with lower rate of context switching. Its basic design is very simple but nevertheless it was designed by an expert in this field, two characteristics which combine to make it attractive to this case. The problem addressed by the GIL has always been *scheduling* threads to the interpreter, not just controlling access to it, and therefore the GIL, a lock implemented as a simple semaphore was the wrong solution. The patches by Antoine and David essentially evolve the GIL into a scheduler, however both cause thread starvation or high rate of context switching in some scenarios: With Floren't write test (http://bugs.python.org/issue7946#msg101120): 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context switching shoots up to 200K/s. 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter patch starves one of the bg threads. 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads starved, context switching shoots up to 250K/s. With UDP test (http://bugs.python.org/file16316/udp-iotest.py), add zlib.compress(b'GIL') to the workload: both gilinter and PyCon patches starve the IO thread. The BFS patch currently involves more overhead by reading the time stamp on each yield and schedule operations. In addition it still remains to address some issues related to timestamps such as getting different time stamp readings on different cores on some (older) multi-core systems. Any thoughts? Nir On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou wrote: > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Mon Apr 12 03:26:49 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Sun, 11 Apr 2010 21:26:49 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: Message-ID: Nir, Per the POSIX standard, both pthread_cond_wait() and pthread_cond_timedwait() need to be performed in a loop. See the fourth paragraph of the description from: > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait > .html For the Windows side, I think you have a similar problem. Condition variables are signaling mechanisms, and so they have a separate boolean predicate associated with them. If you release the mutex that protects the predicate, then after you reacquire the mutex, you have to reevaluate the predicate to ensure that the condition has actually been met. You might want to look at the following for a discussion (not sure how good it is, as I just google?d it quickly) of how to implement POSIX semantics on Windows: > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html Before you can evaluate the effectiveness of any of the proposed scheduling schemes, the fundamental uses of mutexes and condition variables, and their implementations, must be sound. -peter On 4/11/10 6:50 PM, "Nir Aides" wrote: > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt > > I submitted bfs.patch at?http://bugs.python.org/issue7946. It is work in > progress but is ready for some opinion. > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching.?Its basic design is very simple but nevertheless it was designed by > an expert in this field, two?characteristics?which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to the > interpreter, not just controlling access to it, and therefore the GIL, a lock > implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a scheduler, > however both cause thread starvation or high rate of context switching in some > scenarios: > > With Floren't write test (http://bugs.python.org/issue7946#msg101120): > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter patch > starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test (http://bugs.python.org/file16316/udp-iotest.py), add > zlib.compress(b'GIL') to the workload: > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou wrote: >> >> Hello, >> >> As some of you may know, Dave Beazley recently exhibited a situation >> where the new GIL shows quite a poor behaviour (the old GIL isn't very >> good either, but still a little better). This issue is followed in >> http://bugs.python.org/issue7946 >> >> This situation is when an IO-bound thread wants to process a lot of >> incoming packets, while one (or several) CPU-bound thread is also >> running. Each time the IO-bound thread releases the GIL, the CPU-bound >> thread gets it and keeps holding it for at least 5 milliseconds >> (default setting), which limits the number of individual packets which >> can be recv()'ed and processed per second. >> >> I have proposed two mechanisms, based on the same idea: IO-bound >> threads should be able to steal the GIL very quickly, rather than >> having to wait for the whole "thread switching interval" (again, 5 ms >> by default). They differ in how they detect an "IO-bound threads": >> >> - the first mechanism is actually the same mechanism which was >> ?embodied in the original new GIL patch before being removed. In this >> ?approach, IO methods (such as socket.read() in socketmodule.c) >> ?releasing the GIL must use a separate C macro when trying to get the >> ?GIL back again. >> >> - the second mechanism dynamically computes the "interactiveness" of a >> ?thread and allows interactive threads to steal the GIL quickly. In >> ?this approach, IO methods don't have to be modified at all. >> >> Both approaches show similar benchmark results (for the benchmarks >> that I know of) and basically fix the issue put forward by Dave Beazley. >> >> Any thoughts? >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Apr 11 12:43:14 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Apr 2010 22:43:14 +1200 Subject: [Python-Dev] Traceback object has no __class__? Message-ID: <4BC1A7C2.4090003@canterbury.ac.nz> I thought type-class unification was supposed to mean that all objects now have a __class__ attribute. But traceback objects don't seem to: import sys try: raise ValueError except ValueError: tb = sys.exc_info()[2] print tb print tb.__class__ results in: % python2.6 traceback_class.py Traceback (most recent call last): File "traceback_class.py", line 8, in print tb.__class__ AttributeError: __class__ -- Greg From ncoghlan at gmail.com Mon Apr 12 08:56:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Apr 2010 16:56:18 +1000 Subject: [Python-Dev] Traceback object has no __class__? In-Reply-To: <4BC1A7C2.4090003@canterbury.ac.nz> References: <4BC1A7C2.4090003@canterbury.ac.nz> Message-ID: <4BC2C412.2060306@gmail.com> Greg Ewing wrote: > I thought type-class unification was supposed to mean > that all objects now have a __class__ attribute. But > traceback objects don't seem to: > > import sys > > try: > raise ValueError > except ValueError: > tb = sys.exc_info()[2] > print tb > print tb.__class__ I'm not sure what build you're getting that behaviour on, but my svn build of 2.6 has a __class__ attribute for traceback objects, as does the kubuntu 9.10 python 2.6.4 system install. >>> try: ... raise ValueError ... except: ... import sys ... tb = sys.exc_info()[2] ... print type(tb) ... print dir(tb) ... print tb.__class__ ... ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'tb_frame', 'tb_lasti', 'tb_lineno', 'tb_next'] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From willian.paixaoo at gmail.com Mon Apr 12 13:16:10 2010 From: willian.paixaoo at gmail.com (=?UTF-8?Q?Willian_Sodr=C3=A9_da_Paix=C3=A3o?=) Date: Mon, 12 Apr 2010 08:16:10 -0300 Subject: [Python-Dev] Python and compilers In-Reply-To: <4BBE4FE0.8020605@canterbury.ac.nz> References: <2846531d5737614a4248198f77fe55d9@ufpa.br> <4BBB22E7.6060209@voidspace.org.uk> <4BBBACD4.4000906@canterbury.ac.nz> <4BBBE0D8.5040200@gmail.com> <4BBE4FE0.8020605@canterbury.ac.nz> Message-ID: Thank you all. My decision at the moment is a compiler for PICs to accept a language more like python as possible instead of C. How is a college project "solo", I can guarantee at least the first part, a lexical analyzer. Ass: Wil ('.') From nir at winpdb.org Mon Apr 12 14:12:37 2010 From: nir at winpdb.org (Nir Aides) Date: Mon, 12 Apr 2010 15:12:37 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: Message-ID: Hi Peter, There is no need for a loop in bfs_yield(). On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante wrote: > Nir, > > Per the POSIX standard, both pthread_cond_wait() and > pthread_cond_timedwait() need to be performed in a loop. See the fourth > paragraph of the description from: > > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html > > > For the Windows side, I think you have a similar problem. Condition > variables are signaling mechanisms, and so they have a separate boolean > predicate associated with them. If you release the mutex that protects the > predicate, then after you reacquire the mutex, you have to reevaluate the > predicate to ensure that the condition has actually been met. > > You might want to look at the following for a discussion (not sure how good > it is, as I just google?d it quickly) of how to implement POSIX semantics on > Windows: > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > Before you can evaluate the effectiveness of any of the proposed scheduling > schemes, the fundamental uses of mutexes and condition variables, and their > implementations, must be sound. > > -peter > > > > On 4/11/10 6:50 PM, "Nir Aides" < nir at winpdb.org> wrote: > > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt > > I submitted bfs.patch at http://bugs.python.org/issue7946. It is work in > progress but is ready for some opinion. > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed > by an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it, and therefore the GIL, a > lock implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a > scheduler, however both cause thread starvation or high rate of context > switching in some scenarios: > > With Floren't write test ( http://bugs.python.org/issue7946#msg101120): > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter > patch starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test ( http://bugs.python.org/file16316/udp-iotest.py), add > zlib.compress(b'GIL') to the workload: > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < solipsis at pitrou.net> > wrote: > > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ______________________________ _________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Mon Apr 12 14:37:59 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Mon, 12 Apr 2010 08:37:59 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: Message-ID: Hmm, so I see in bfs_yield(): + if (tstate != NULL && bfs_thread_switch == tstate) { + COND_RESET(tstate->bfs_cond); + COND_WAIT(tstate->bfs_cond, bfs_mutex); + } So, to me, the above code says, ?Wait for the condition that tstate is either NULL, or bfs_thread_switch does not equal tstate?. So the predicate is: ?(tstate != NULL && bfs_thread_switch == tstate)?. If the predicate is True before you call COND_WAIT() and True after you call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you need to loop until the predicate is False. There is no guarantee that a condition wait actually did anything at all. Yes, there will be spurious wake ups, but you can?t tell if the thread actually blocked and then woke up, or never blocked at all. If it never actually blocks, then that code path is not helpful. On Windows, before this loop in bfs_schedule(): + COND_RESET(tstate->bfs_cond); + while (bfs_thread != tstate) { + _bfs_timed_wait(tstate, timestamp); + timestamp = get_timestamp(); + } You might want to avoid the call to reset the condition variable if the predicate is already False. -peter On 4/12/10 8:12 AM, "Nir Aides" wrote: > Hi Peter, > > There is no need for a loop in bfs_yield().? > > > On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante > wrote: >> Nir, >> >> Per the POSIX standard, both pthread_cond_wait() and pthread_cond_timedwait() >> need to be performed in a loop. ?See the fourth paragraph of the description >> from: >> >>> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwa >>> it.html >>> >> ait.html> >> >> For the Windows side, I think you have a similar problem. Condition variables >> are signaling mechanisms, and so they have a separate boolean predicate >> associated with them. If you release the mutex that protects the predicate, >> then after you reacquire the mutex, you have to reevaluate the predicate to >> ensure that the condition has actually been met. >> >> You might want to look at the following for a discussion (not sure how good >> it is, as I just google?d it quickly) of how to implement POSIX semantics on >> Windows: >> >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> >> >> Before you can evaluate the effectiveness of any of the proposed scheduling >> schemes, the fundamental uses of mutexes and condition variables, and their >> implementations, must be sound. >> >> -peter >> >> >> >> On 4/11/10 6:50 PM, "Nir Aides" < nir at winpdb.org > >> wrote: >> >>> Hello all, >>> >>> I would like to kick this discussion back to life with a simplified >>> implementation of the BFS scheduler, designed by the Linux kernel hacker Con >>> Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt >>> >>> >>> I submitted bfs.patch at? http://bugs.python.org/issue7946 >>> . It is work in progress but is ready for >>> some opinion. >>> >>> On my machine BFS gives comparable performance to gilinter, and seems to >>> schedule threads more fairly, predictably, and with lower rate of context >>> switching.?Its basic design is very simple but nevertheless it was designed >>> by an expert in this field, two?characteristics?which combine to make it >>> attractive to this case. >>> >>> The problem addressed by the GIL has always been *scheduling* threads to the >>> interpreter, not just controlling access to it, and therefore the GIL, a >>> lock implemented as a simple semaphore was the wrong solution. >>> >>> The patches by Antoine and David essentially evolve the GIL into a >>> scheduler, however both cause thread starvation or high rate of context >>> switching in some scenarios: >>> >>> With Floren't write test ( http://bugs.python.org/issue7946#msg101120 >>> ): >>> 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context >>> switching shoots up to 200K/s. >>> 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter patch >>> starves one of the bg threads. >>> 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads >>> starved, context switching shoots up to 250K/s. >>> >>> With UDP test ( http://bugs.python.org/file16316/udp-iotest.py >>> ), add >>> zlib.compress(b'GIL') to the workload: >>> both gilinter and PyCon patches starve the IO thread. >>> >>> The BFS patch currently involves more overhead by reading the time stamp on >>> each yield and schedule operations. In addition it still remains to address >>> some issues related to timestamps such as getting different time stamp >>> readings on different cores on some (older) multi-core systems. >>> >>> Any thoughts? >>> >>> Nir >>> >>> >>> >>> On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < solipsis at pitrou.net >>> > wrote: >>>> >>>> Hello, >>>> >>>> As some of you may know, Dave Beazley recently exhibited a situation >>>> where the new GIL shows quite a poor behaviour (the old GIL isn't very >>>> good either, but still a little better). This issue is followed in >>>> http://bugs.python.org/issue7946 >>>> >>>> This situation is when an IO-bound thread wants to process a lot of >>>> incoming packets, while one (or several) CPU-bound thread is also >>>> running. Each time the IO-bound thread releases the GIL, the CPU-bound >>>> thread gets it and keeps holding it for at least 5 milliseconds >>>> (default setting), which limits the number of individual packets which >>>> can be recv()'ed and processed per second. >>>> >>>> I have proposed two mechanisms, based on the same idea: IO-bound >>>> threads should be able to steal the GIL very quickly, rather than >>>> having to wait for the whole "thread switching interval" (again, 5 ms >>>> by default). They differ in how they detect an "IO-bound threads": >>>> >>>> - the first mechanism is actually the same mechanism which was >>>> ??embodied in the original new GIL patch before being removed. In this >>>> ??approach, IO methods (such as socket.read() in socketmodule.c) >>>> ??releasing the GIL must use a separate C macro when trying to get the >>>> ??GIL back again. >>>> >>>> - the second mechanism dynamically computes the "interactiveness" of a >>>> ??thread and allows interactive threads to steal the GIL quickly. In >>>> ??this approach, IO methods don't have to be modified at all. >>>> >>>> Both approaches show similar benchmark results (for the benchmarks >>>> that I know of) and basically fix the issue put forward by Dave Beazley. >>>> >>>> Any thoughts? >>>> >>>> Regards >>>> >>>> Antoine. >>>> >>>> >>>> ______________________________ _________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> http://mail.python.org/mailman/listinfo/python-dev >>>> >>>> Unsubscribe: >>>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org >>>> >>> >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> http://mail.python.org/mailman/listinfo/python-dev >>> >>> Unsubscribe: >>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.c >>> om >>> >> com> > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Mon Apr 12 14:48:25 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Mon, 12 Apr 2010 08:48:25 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: Message-ID: And why the for(;;) loop in bfs_schedule()? I don?t see a code path that would loop there. Perhaps I am missing it ... -peter On 4/12/10 8:37 AM, "Peter Portante" wrote: > Hmm, so I see in bfs_yield(): > > + if (tstate != NULL && bfs_thread_switch == tstate) { > + COND_RESET(tstate->bfs_cond); > + COND_WAIT(tstate->bfs_cond, bfs_mutex); > + } > > So, to me, the above code says, ?Wait for the condition that tstate is either > NULL, or bfs_thread_switch does not equal tstate?. So the predicate is: > ?(tstate != NULL && bfs_thread_switch == tstate)?. > > If the predicate is True before you call COND_WAIT() and True after you call > COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you need to > loop until the predicate is False. There is no guarantee that a condition wait > actually did anything at all. Yes, there will be spurious wake ups, but you > can?t tell if the thread actually blocked and then woke up, or never blocked > at all. If it never actually blocks, then that code path is not helpful. > > On Windows, before this loop in bfs_schedule(): > > + COND_RESET(tstate->bfs_cond); > + while (bfs_thread != tstate) { > + _bfs_timed_wait(tstate, timestamp); > + timestamp = get_timestamp(); > + } > > You might want to avoid the call to reset the condition variable if the > predicate is already False. > > -peter > > > On 4/12/10 8:12 AM, "Nir Aides" wrote: > >> Hi Peter, >> >> There is no need for a loop in bfs_yield().? >> >> >> On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante >> wrote: >>> Nir, >>> >>> Per the POSIX standard, both pthread_cond_wait() and >>> pthread_cond_timedwait() need to be performed in a loop. ?See the fourth >>> paragraph of the description from: >>> >>>> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedw >>>> ait.html >>>> >>> wait.html> >>> >>> For the Windows side, I think you have a similar problem. Condition >>> variables are signaling mechanisms, and so they have a separate boolean >>> predicate associated with them. If you release the mutex that protects the >>> predicate, then after you reacquire the mutex, you have to reevaluate the >>> predicate to ensure that the condition has actually been met. >>> >>> You might want to look at the following for a discussion (not sure how good >>> it is, as I just google?d it quickly) of how to implement POSIX semantics on >>> Windows: >>> >>>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>> >>> >>> Before you can evaluate the effectiveness of any of the proposed scheduling >>> schemes, the fundamental uses of mutexes and condition variables, and their >>> implementations, must be sound. >>> >>> -peter >>> >>> >>> >>> On 4/11/10 6:50 PM, "Nir Aides" < nir at winpdb.org > >>> wrote: >>> >>>> Hello all, >>>> >>>> I would like to kick this discussion back to life with a simplified >>>> implementation of the BFS scheduler, designed by the Linux kernel hacker >>>> Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt >>>> >>>> >>>> I submitted bfs.patch at? http://bugs.python.org/issue7946 >>>> . It is work in progress but is ready >>>> for some opinion. >>>> >>>> On my machine BFS gives comparable performance to gilinter, and seems to >>>> schedule threads more fairly, predictably, and with lower rate of context >>>> switching.?Its basic design is very simple but nevertheless it was designed >>>> by an expert in this field, two?characteristics?which combine to make it >>>> attractive to this case. >>>> >>>> The problem addressed by the GIL has always been *scheduling* threads to >>>> the interpreter, not just controlling access to it, and therefore the GIL, >>>> a lock implemented as a simple semaphore was the wrong solution. >>>> >>>> The patches by Antoine and David essentially evolve the GIL into a >>>> scheduler, however both cause thread starvation or high rate of context >>>> switching in some scenarios: >>>> >>>> With Floren't write test ( http://bugs.python.org/issue7946#msg101120 >>>> ): >>>> 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context >>>> switching shoots up to 200K/s. >>>> 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter >>>> patch starves one of the bg threads. >>>> 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads >>>> starved, context switching shoots up to 250K/s. >>>> >>>> With UDP test ( http://bugs.python.org/file16316/udp-iotest.py >>>> ), add >>>> zlib.compress(b'GIL') to the workload: >>>> both gilinter and PyCon patches starve the IO thread. >>>> >>>> The BFS patch currently involves more overhead by reading the time stamp on >>>> each yield and schedule operations. In addition it still remains to address >>>> some issues related to timestamps such as getting different time stamp >>>> readings on different cores on some (older) multi-core systems. >>>> >>>> Any thoughts? >>>> >>>> Nir >>>> >>>> >>>> >>>> On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < solipsis at pitrou.net >>>> > wrote: >>>>> >>>>> Hello, >>>>> >>>>> As some of you may know, Dave Beazley recently exhibited a situation >>>>> where the new GIL shows quite a poor behaviour (the old GIL isn't very >>>>> good either, but still a little better). This issue is followed in >>>>> http://bugs.python.org/issue7946 >>>>> >>>>> This situation is when an IO-bound thread wants to process a lot of >>>>> incoming packets, while one (or several) CPU-bound thread is also >>>>> running. Each time the IO-bound thread releases the GIL, the CPU-bound >>>>> thread gets it and keeps holding it for at least 5 milliseconds >>>>> (default setting), which limits the number of individual packets which >>>>> can be recv()'ed and processed per second. >>>>> >>>>> I have proposed two mechanisms, based on the same idea: IO-bound >>>>> threads should be able to steal the GIL very quickly, rather than >>>>> having to wait for the whole "thread switching interval" (again, 5 ms >>>>> by default). They differ in how they detect an "IO-bound threads": >>>>> >>>>> - the first mechanism is actually the same mechanism which was >>>>> ??embodied in the original new GIL patch before being removed. In this >>>>> ??approach, IO methods (such as socket.read() in socketmodule.c) >>>>> ??releasing the GIL must use a separate C macro when trying to get the >>>>> ??GIL back again. >>>>> >>>>> - the second mechanism dynamically computes the "interactiveness" of a >>>>> ??thread and allows interactive threads to steal the GIL quickly. In >>>>> ??this approach, IO methods don't have to be modified at all. >>>>> >>>>> Both approaches show similar benchmark results (for the benchmarks >>>>> that I know of) and basically fix the issue put forward by Dave Beazley. >>>>> >>>>> Any thoughts? >>>>> >>>>> Regards >>>>> >>>>> Antoine. >>>>> >>>>> >>>>> ______________________________ _________________ >>>>> Python-Dev mailing list >>>>> Python-Dev at python.org >>>>> http://mail.python.org/mailman/listinfo/python-dev >>>>> >>>>> Unsubscribe: >>>>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> http://mail.python.org/mailman/listinfo/python-dev >>>> >>>> Unsubscribe: >>>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail. >>>> com >>>> >>> .com> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.co>> m -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Mon Apr 12 15:07:21 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 12 Apr 2010 16:07:21 +0300 Subject: [Python-Dev] [RELEASED] 2.7 beta 1 In-Reply-To: References: Message-ID: On Sun, Apr 11, 2010 at 1:13 AM, average wrote: > > There are so many features taken from 3.0 that I fear that it will > postpone its adoption interminably (it is, in practice, treated as > "beta" software itself). ?By making it doctrine that it won't be > official until the next "major" Python release, it will encourage > those who are able, to just make the jump to 3.0, while those who > cannot will have the subtle pressure to make the shift, however > gradual. > Additionally, it will give the community further incentive > to make Python3 all that it was intended to be. ?Personally, the > timing of v3 prevented me from fully participating in that effort, > and, not ignoring the work of those who did contribute, I think many > of us feel that it has not reached its potential. The same problem. For me it was possible to participate in standard library development only after Python Alphas with Windows binaries were released. I could test both new features and old bugs. Having a requirement that every developer should be able to compile binaries has an adverse effect on the quality of standard library. The absence of public Roadmap also makes it hard to judge the aforementioned "desired potential". It could be possible to compile a public list like http://dungeonhack.sourceforge.net/Roadmap I am afraid of two things with forthcoming Python releases. 1. feature creeping 2. feature missing And an overview of Python development in the form of release timer and roadmap will remove the remnants of fear and uncertainty and surely attract new people for sprints. Regardless of said above it is great to feel the hard work behind the scenes that makes new releases popping up. Thanks. -- anatoly t. From abpillai at gmail.com Mon Apr 12 15:36:29 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Mon, 12 Apr 2010 19:06:29 +0530 Subject: [Python-Dev] OS information, tags Message-ID: Hi, I am surprised to see that the bug-tracker doesn't have an OS classifier or ability to add tags ? Since a number of issues reported seem to be OS specific (one can find a lot of Windows only issues upon a search), won't adding these fields help bug triaging ? I am not sure which software is being used by bug tracker so excuse me if this has been already discussed in this form. A quick search in my gmail archives yielded no such discussion. Thanks, -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From orsenthil at gmail.com Mon Apr 12 15:51:51 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 12 Apr 2010 19:21:51 +0530 Subject: [Python-Dev] OS information, tags In-Reply-To: References: Message-ID: <20100412135151.GA31685@remy> On Mon, Apr 12, 2010 at 07:06:29PM +0530, Anand Balachandran Pillai wrote: > ?????????? I am surprised to see that the bug-tracker > doesn't have an OS classifier or ability to add > tags ? Since a number of issues reported seem to There is one. In the Components you can do a multiple select and it has Macintosh , Windows as options. > I am not sure which software is being used by > bug tracker so excuse me if this has been already > discussed in this form. A quick search in my gmail > archives yielded no such discussion. It is Roundup developed by Richard (Pygame ??). There is tracker bug list too, which you can find referenced in the bugs.python.org page. -- Senthil "Elves and Dragons!" I says to him. "Cabbages and potatoes are better for you and me." -- J. R. R. Tolkien From rdmurray at bitdance.com Mon Apr 12 16:44:06 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 12 Apr 2010 10:44:06 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: References: Message-ID: <20100412144406.940921FF82E@kimball.webabinitio.net> On Mon, 12 Apr 2010 19:06:29 +0530, Anand Balachandran Pillai wrote: > I am surprised to see that the bug-tracker > doesn't have an OS classifier or ability to add > tags ? Since a number of issues reported seem to > be OS specific (one can find a lot of Windows only > issues upon a search), won't adding these fields > help bug triaging ? As someone else pointed out, we do have Windows and Mac components. Historically the number of bugs specific to other OSes has been small enough that it hasn't been worth having classifiers for them. Or perhaps it's that we haven't had devs who were specifically tracking those platforms... We can add additional components if the community sees a need for them. -- R. David Murray www.bitdance.com From tjreedy at udel.edu Mon Apr 12 19:13:05 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Apr 2010 13:13:05 -0400 Subject: [Python-Dev] Traceback object has no __class__? In-Reply-To: <4BC1A7C2.4090003@canterbury.ac.nz> References: <4BC1A7C2.4090003@canterbury.ac.nz> Message-ID: On 4/11/2010 6:43 AM, Greg Ewing wrote: > import sys > > try: > raise ValueError > except ValueError: > tb = sys.exc_info()[2] > print tb > print tb.__class__ # 3.1.2 From nir at winpdb.org Mon Apr 12 21:17:58 2010 From: nir at winpdb.org (Nir Aides) Date: Mon, 12 Apr 2010 22:17:58 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: Message-ID: The loop-less wait is similar to the one in new GIL. It is used to force a switch to next thread in particular scenario and the motivation is explained in comment to another if clause a few lines up. Those two if clauses can be joined though. On Mon, Apr 12, 2010 at 3:37 PM, Peter Portante wrote: > Hmm, so I see in bfs_yield(): > > + if (tstate != NULL && bfs_thread_switch == tstate) { > + COND_RESET(tstate->bfs_cond); > + COND_WAIT(tstate->bfs_cond, bfs_mutex); > + } > > So, to me, the above code says, ?Wait for the condition that tstate is > either NULL, or bfs_thread_switch does not equal tstate?. So the predicate > is: ?(tstate != NULL && bfs_thread_switch == tstate)?. > > If the predicate is True before you call COND_WAIT() and True after you > call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you > need to loop until the predicate is False. There is no guarantee that a > condition wait actually did anything at all. Yes, there will be spurious > wake ups, but you can?t tell if the thread actually blocked and then woke > up, or never blocked at all. If it never actually blocks, then that code > path is not helpful. > > On Windows, before this loop in bfs_schedule(): > > + COND_RESET(tstate->bfs_cond); > + while (bfs_thread != tstate) { > + _bfs_timed_wait(tstate, timestamp); > + timestamp = get_timestamp(); > + } > > You might want to avoid the call to reset the condition variable if the > predicate is already False. > > -peter > > > > On 4/12/10 8:12 AM, "Nir Aides" wrote: > > Hi Peter, > > There is no need for a loop in bfs_yield(). > > > On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante < > peter.a.portante at gmail.com> wrote: > > Nir, > > Per the POSIX standard, both pthread_cond_wait() and > pthread_cond_timedwait() need to be performed in a loop. See the fourth > paragraph of the description from: > > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html< > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html> > > > > For the Windows side, I think you have a similar problem. Condition > variables are signaling mechanisms, and so they have a separate boolean > predicate associated with them. If you release the mutex that protects the > predicate, then after you reacquire the mutex, you have to reevaluate the > predicate to ensure that the condition has actually been met. > > You might want to look at the following for a discussion (not sure how good > it is, as I just google?d it quickly) of how to implement POSIX semantics on > Windows: > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html < > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html> > > > Before you can evaluate the effectiveness of any of the proposed scheduling > schemes, the fundamental uses of mutexes and condition variables, and their > implementations, must be sound. > > -peter > > > > On 4/11/10 6:50 PM, "Nir Aides" < > nir at winpdb.org > wrote: > > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt < > http://ck.kolivas.org/patches/bfs/sched-BFS.txt> > > I submitted bfs.patch at http://bugs.python.org/issue7946 < > http://bugs.python.org/issue7946> . It is work in progress but is ready > for some opinion. > > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed > by an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it, and therefore the GIL, a > lock implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a > scheduler, however both cause thread starvation or high rate of context > switching in some scenarios: > > With Floren't write test ( http://bugs.python.org/issue7946#msg101120 < > http://bugs.python.org/issue7946#msg101120> ): > > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter > patch starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test ( http://bugs.python.org/file16316/udp-iotest.py < > http://bugs.python.org/file16316/udp-iotest.py> ), add > zlib.compress(b'GIL') to the workload: > > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < > solipsis at pitrou.net > wrote: > > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ______________________________ > _________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org < > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org> > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com< > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com> > > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nir at winpdb.org Mon Apr 12 21:19:43 2010 From: nir at winpdb.org (Nir Aides) Date: Mon, 12 Apr 2010 22:19:43 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: Message-ID: At some point there was a loop, later it remained since I feel it is more readable than a bunch of nested if-else clauses. Should probably replace with do {...} while(0) I was conditioned with electrical shocks in the dungeons of a corporate to always use for loops. I uploaded the patch to Rietveld for code review comments: http://codereview.appspot.com/857049 Nir On Mon, Apr 12, 2010 at 3:48 PM, Peter Portante wrote: > And why the for(;;) loop in bfs_schedule()? I don?t see a code path that > would loop there. Perhaps I am missing it ... > > -peter > > > > On 4/12/10 8:37 AM, "Peter Portante" wrote: > > Hmm, so I see in bfs_yield(): > > + if (tstate != NULL && bfs_thread_switch == tstate) { > + COND_RESET(tstate->bfs_cond); > + COND_WAIT(tstate->bfs_cond, bfs_mutex); > + } > > So, to me, the above code says, ?Wait for the condition that tstate is > either NULL, or bfs_thread_switch does not equal tstate?. So the predicate > is: ?(tstate != NULL && bfs_thread_switch == tstate)?. > > If the predicate is True before you call COND_WAIT() and True after you > call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you > need to loop until the predicate is False. There is no guarantee that a > condition wait actually did anything at all. Yes, there will be spurious > wake ups, but you can?t tell if the thread actually blocked and then woke > up, or never blocked at all. If it never actually blocks, then that code > path is not helpful. > > On Windows, before this loop in bfs_schedule(): > > + COND_RESET(tstate->bfs_cond); > + while (bfs_thread != tstate) { > + _bfs_timed_wait(tstate, timestamp); > + timestamp = get_timestamp(); > + } > > You might want to avoid the call to reset the condition variable if the > predicate is already False. > > -peter > > > On 4/12/10 8:12 AM, "Nir Aides" wrote: > > Hi Peter, > > There is no need for a loop in bfs_yield(). > > > On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante < > peter.a.portante at gmail.com> wrote: > > Nir, > > Per the POSIX standard, both pthread_cond_wait() and > pthread_cond_timedwait() need to be performed in a loop. See the fourth > paragraph of the description from: > > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html< > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html> > > > > For the Windows side, I think you have a similar problem. Condition > variables are signaling mechanisms, and so they have a separate boolean > predicate associated with them. If you release the mutex that protects the > predicate, then after you reacquire the mutex, you have to reevaluate the > predicate to ensure that the condition has actually been met. > > You might want to look at the following for a discussion (not sure how good > it is, as I just google?d it quickly) of how to implement POSIX semantics on > Windows: > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html < > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html> > > > Before you can evaluate the effectiveness of any of the proposed scheduling > schemes, the fundamental uses of mutexes and condition variables, and their > implementations, must be sound. > > -peter > > > > On 4/11/10 6:50 PM, "Nir Aides" < nir at winpdb.org > > wrote: > > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt < > http://ck.kolivas.org/patches/bfs/sched-BFS.txt> > > I submitted bfs.patch at http://bugs.python.org/issue7946 < > http://bugs.python.org/issue7946> . It is work in progress but is ready > for some opinion. > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed > by an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it, and therefore the GIL, a > lock implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a > scheduler, however both cause thread starvation or high rate of context > switching in some scenarios: > > With Floren't write test ( http://bugs.python.org/issue7946#msg101120 < > http://bugs.python.org/issue7946#msg101120> ): > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter > patch starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test ( http://bugs.python.org/file16316/udp-iotest.py < > http://bugs.python.org/file16316/udp-iotest.py> ), add > zlib.compress(b'GIL') to the workload: > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < solipsis at pitrou.net < > http://solipsis at pitrou.net> > wrote: > > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ______________________________ _________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org < > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org> > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com< > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com> > > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Mon Apr 12 21:25:21 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Mon, 12 Apr 2010 15:25:21 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: Message-ID: Yes, but unless you loop until the predicate is False, no forced switch is guaranteed to occur. You might as well remove the code. If you want to keep the code as is, call me when you need a life guard to help debug mystifying behaviors. ;) -peter On 4/12/10 3:17 PM, "Nir Aides" wrote: > The loop-less wait is similar to the one in new GIL. It is used to force a > switch to next thread in particular scenario and the motivation is explained > in comment to another if clause a few lines up. Those two if clauses can be > joined though. > > > On Mon, Apr 12, 2010 at 3:37 PM, Peter Portante > wrote: >> Hmm, so I see in bfs_yield(): >> >> + ???if (tstate != NULL && bfs_thread_switch == tstate) { >> + ???????COND_RESET(tstate->bfs_cond); >> + ???????COND_WAIT(tstate->bfs_cond, bfs_mutex); >> + ???} >> >> So, to me, the above code says, ?Wait for the condition that tstate is either >> NULL, or bfs_thread_switch does not equal tstate?. So the predicate is: >> ?(tstate != NULL && bfs_thread_switch == tstate)?. >> >> If the predicate is True before you call COND_WAIT() and True after you call >> COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you need to >> loop until the predicate is False. There is no guarantee that a condition >> wait actually did anything at all. Yes, there will be spurious wake ups, but >> you can?t tell if the thread actually blocked and then woke up, or never >> blocked at all. If it never actually blocks, then that code path is not >> helpful. >> >> On Windows, before this loop in bfs_schedule(): >> >> + ???????COND_RESET(tstate->bfs_cond); >> + ???????while (bfs_thread != tstate) { >> + ???????????_bfs_timed_wait(tstate, timestamp); >> + ???????????timestamp = get_timestamp(); >> + ???????} >> >> You might want to avoid the call to reset the condition variable if the >> predicate is already False. >> >> -peter >> >> >> >> On 4/12/10 8:12 AM, "Nir Aides" > >> wrote: >> >>> Hi Peter, >>> >>> There is no need for a loop in bfs_yield().? >>> >>> >>> On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante >> > wrote: >>>> Nir, >>>> >>>> Per the POSIX standard, both pthread_cond_wait() and >>>> pthread_cond_timedwait() need to be performed in a loop. ?See the fourth >>>> paragraph of the description from: >>>> >>>>> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timed >>>>> wait.html >>>>> >>>> dwait.html> >>>> >>>> For the Windows side, I think you have a similar problem. Condition >>>> variables are signaling mechanisms, and so they have a separate boolean >>>> predicate associated with them. If you release the mutex that protects the >>>> predicate, then after you reacquire the mutex, you have to reevaluate the >>>> predicate to ensure that the condition has actually been met. >>>> >>>> You might want to look at the following for a discussion (not sure how good >>>> it is, as I just google?d it quickly) of how to implement POSIX semantics >>>> on Windows: >>>> >>>>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>> >>>> >>>> Before you can evaluate the effectiveness of any of the proposed scheduling >>>> schemes, the fundamental uses of mutexes and condition variables, and their >>>> implementations, must be sound. >>>> >>>> -peter >>>> >>>> >>>> >>>> On 4/11/10 6:50 PM, "Nir Aides" < >>>> nir at winpdb.org > wrote: >>>> >>>>> Hello all, >>>>> >>>>> I would like to kick this discussion back to life with a simplified >>>>> implementation of the BFS scheduler, designed by the Linux kernel hacker >>>>> Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt >>>>> >>>>> >>>>> I submitted bfs.patch at? http://bugs.python.org/issue7946 >>>>> . It is work in progress but is ready >>>>> for some opinion. >>>>> >>>>> >>>>> On my machine BFS gives comparable performance to gilinter, and seems to >>>>> schedule threads more fairly, predictably, and with lower rate of context >>>>> switching.?Its basic design is very simple but nevertheless it was >>>>> designed by an expert in this field, two?characteristics?which combine to >>>>> make it attractive to this case. >>>>> >>>>> The problem addressed by the GIL has always been *scheduling* threads to >>>>> the interpreter, not just controlling access to it, and therefore the GIL, >>>>> a lock implemented as a simple semaphore was the wrong solution. >>>>> >>>>> The patches by Antoine and David essentially evolve the GIL into a >>>>> scheduler, however both cause thread starvation or high rate of context >>>>> switching in some scenarios: >>>>> >>>>> With Floren't write test ( http://bugs.python.org/issue7946#msg101120 >>>>> ): >>>>> >>>>> 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context >>>>> switching shoots up to 200K/s. >>>>> 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter >>>>> patch starves one of the bg threads. >>>>> 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads >>>>> starved, context switching shoots up to 250K/s. >>>>> >>>>> With UDP test ( http://bugs.python.org/file16316/udp-iotest.py >>>>> ), add >>>>> zlib.compress(b'GIL') to the workload: >>>>> >>>>> both gilinter and PyCon patches starve the IO thread. >>>>> >>>>> The BFS patch currently involves more overhead by reading the time stamp >>>>> on each yield and schedule operations. In addition it still remains to >>>>> address some issues related to timestamps such as getting different time >>>>> stamp readings on different cores on some (older) multi-core systems. >>>>> >>>>> Any thoughts? >>>>> >>>>> Nir >>>>> >>>>> >>>>> >>>>> On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < >>>>> solipsis at pitrou.net >>>>> > wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> As some of you may know, Dave Beazley recently exhibited a situation >>>>>> where the new GIL shows quite a poor behaviour (the old GIL isn't very >>>>>> good either, but still a little better). This issue is followed in >>>>>> http://bugs.python.org/issue7946 >>>>>> >>>>>> This situation is when an IO-bound thread wants to process a lot of >>>>>> incoming packets, while one (or several) CPU-bound thread is also >>>>>> running. Each time the IO-bound thread releases the GIL, the CPU-bound >>>>>> thread gets it and keeps holding it for at least 5 milliseconds >>>>>> (default setting), which limits the number of individual packets which >>>>>> can be recv()'ed and processed per second. >>>>>> >>>>>> I have proposed two mechanisms, based on the same idea: IO-bound >>>>>> threads should be able to steal the GIL very quickly, rather than >>>>>> having to wait for the whole "thread switching interval" (again, 5 ms >>>>>> by default). They differ in how they detect an "IO-bound threads": >>>>>> >>>>>> - the first mechanism is actually the same mechanism which was >>>>>> ??embodied in the original new GIL patch before being removed. In this >>>>>> ??approach, IO methods (such as socket.read() in socketmodule.c) >>>>>> ??releasing the GIL must use a separate C macro when trying to get the >>>>>> ??GIL back again. >>>>>> >>>>>> - the second mechanism dynamically computes the "interactiveness" of a >>>>>> ??thread and allows interactive threads to steal the GIL quickly. In >>>>>> ??this approach, IO methods don't have to be modified at all. >>>>>> >>>>>> Both approaches show similar benchmark results (for the benchmarks >>>>>> that I know of) and basically fix the issue put forward by Dave Beazley. >>>>>> >>>>>> Any thoughts? >>>>>> >>>>>> Regards >>>>>> >>>>>> Antoine. >>>>>> >>>>>> >>>>>> ______________________________ >>>>>> _________________ >>>>>> Python-Dev mailing list >>>>>> Python-Dev at python.org >>>>>> >>>>>> http://mail.python.org/mailman/listinfo/python-dev >>>>>> >>>>>> Unsubscribe: >>>>>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org >>>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Python-Dev mailing list >>>>> Python-Dev at python.org >>>>> >>>>> http://mail.python.org/mailman/listinfo/python-dev >>>>> >>>>> Unsubscribe: >>>>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail >>>>> .com >>>>> >>>> l.com> >>> >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> http://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.c >>> om > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nir at winpdb.org Mon Apr 12 21:36:51 2010 From: nir at winpdb.org (Nir Aides) Date: Mon, 12 Apr 2010 22:36:51 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: Message-ID: Please describe clearly a step by step scenario in which that code will fail. On Mon, Apr 12, 2010 at 10:25 PM, Peter Portante wrote: > Yes, but unless you loop until the predicate is False, no forced switch > is guaranteed to occur. You might as well remove the code. If you want to > keep the code as is, call me when you need a life guard to help debug > mystifying behaviors. ;) -peter > > > > On 4/12/10 3:17 PM, "Nir Aides" wrote: > > The loop-less wait is similar to the one in new GIL. It is used to force a > switch to next thread in particular scenario and the motivation is explained > in comment to another if clause a few lines up. Those two if clauses can be > joined though. > > > On Mon, Apr 12, 2010 at 3:37 PM, Peter Portante < > peter.a.portante at gmail.com> wrote: > > Hmm, so I see in > bfs_yield(): > > + if (tstate != NULL && bfs_thread_switch == tstate) { > + COND_RESET(tstate->bfs_cond); > + COND_WAIT(tstate->bfs_cond, bfs_mutex); > + } > > So, to me, the above code says, ?Wait for the condition that tstate is > either NULL, or bfs_thread_switch does not equal tstate?. So the predicate > is: ?(tstate != NULL && bfs_thread_switch == tstate)?. > > If the predicate is True before you call COND_WAIT() and True after you > call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you > need to loop until the predicate is False. There is no guarantee that a > condition wait actually did anything at all. Yes, there will be spurious > wake ups, but you can?t tell if the thread actually blocked and then woke > up, or never blocked at all. If it never actually blocks, then that code > path is not helpful. > > On Windows, before this loop in bfs_schedule(): > > + COND_RESET(tstate->bfs_cond); > + while (bfs_thread != tstate) { > + _bfs_timed_wait(tstate, timestamp); > + timestamp = get_timestamp(); > + } > > You might want to avoid the call to reset the condition variable if the > predicate is already False. > > -peter > > > > On 4/12/10 8:12 AM, "Nir Aides" > > wrote: > > Hi Peter, > > There is no need for a loop in bfs_yield(). > > > On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante < > peter.a.portante at gmail.com > wrote: > > Nir, > > Per the POSIX standard, both pthread_cond_wait() and > pthread_cond_timedwait() need to be performed in a loop. See the fourth > paragraph of the description from: > > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html< > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html> > > > > For the Windows side, I think you have a similar problem. Condition > variables are signaling mechanisms, and so they have a separate boolean > predicate associated with them. If you release the mutex that protects the > predicate, then after you reacquire the mutex, you have to reevaluate the > predicate to ensure that the condition has actually been met. > > You might want to look at the following for a discussion (not sure how good > it is, as I just google?d it quickly) of how to implement POSIX semantics on > Windows: > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html < > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html> > > > Before you can evaluate the effectiveness of any of the proposed scheduling > schemes, the fundamental uses of mutexes and condition variables, and their > implementations, must be sound. > > -peter > > > > On 4/11/10 6:50 PM, "Nir Aides" < > nir at winpdb.org > wrote: > > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt < > http://ck.kolivas.org/patches/bfs/sched-BFS.txt> > > I submitted bfs.patch at http://bugs.python.org/issue7946 < > http://bugs.python.org/issue7946> . It is work in progress but is ready > for some opinion. > > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed > by an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it, and therefore the GIL, a > lock implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a > scheduler, however both cause thread starvation or high rate of context > switching in some scenarios: > > With Floren't write test ( http://bugs.python.org/issue7946#msg101120 < > http://bugs.python.org/issue7946#msg101120> ): > > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter > patch starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test ( http://bugs.python.org/file16316/udp-iotest.py < > http://bugs.python.org/file16316/udp-iotest.py> ), add > zlib.compress(b'GIL') to the workload: > > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < > solipsis at pitrou.net < > http://solipsis at pitrou.net> > wrote: > > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ______________________________ > _________________ > Python-Dev mailing list > Python-Dev at python.org < > http://Python-Dev at python.org> > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org < > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org> > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org < > http://Python-Dev at python.org> > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com< > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com> > > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.a.portante at gmail.com Mon Apr 12 21:49:21 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Mon, 12 Apr 2010 15:49:21 -0400 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: Message-ID: That code will not switch to another thread if the condition variable either does not actually block the thread on the call (which is allowed by the standard to give implementations some flexibility for making it work correctly ? read the standard reasoning for more information), or the thread is woken up without a predicate change (also allowed by the standard for similar reasons). Both of those cases are called ?spurious wake-ups?. You may not be able to readily get your implementation to behavior that way, but in the wild, we need to account for this behavior because Cpython will be run on systems where it will happen. :) -peter On 4/12/10 3:36 PM, "Nir Aides" wrote: > Please describe clearly a step by step scenario in which that code will fail. > > > On Mon, Apr 12, 2010 at 10:25 PM, Peter Portante > wrote: >> Yes, but unless you loop until the predicate is False, no forced switch is >> guaranteed to occur. You might as well remove the code. If you want to keep >> the code as is, call me when you need a life guard to help debug mystifying >> behaviors. ;) -peter >> >> >> >> On 4/12/10 3:17 PM, "Nir Aides" > >> wrote: >> >>> The loop-less wait is similar to the one in new GIL. It is used to force a >>> switch to next thread in particular scenario and the motivation is explained >>> in comment to another if clause a few lines up. Those two if clauses can be >>> joined though. >>> >>> >>> On Mon, Apr 12, 2010 at 3:37 PM, Peter Portante >> > wrote: >>>> Hmm, so I see in >>>> bfs_yield(): >>>> >>>> + ???if (tstate != NULL && bfs_thread_switch == tstate) { >>>> + ???????COND_RESET(tstate->bfs_cond); >>>> + ???????COND_WAIT(tstate->bfs_cond, bfs_mutex); >>>> + ???} >>>> >>>> So, to me, the above code says, ?Wait for the condition that tstate is >>>> either NULL, or bfs_thread_switch does not equal tstate?. So the predicate >>>> is: ?(tstate != NULL && bfs_thread_switch == tstate)?. >>>> >>>> If the predicate is True before you call COND_WAIT() and True after you >>>> call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you >>>> need to loop until the predicate is False. There is no guarantee that a >>>> condition wait actually did anything at all. Yes, there will be spurious >>>> wake ups, but you can?t tell if the thread actually blocked and then woke >>>> up, or never blocked at all. If it never actually blocks, then that code >>>> path is not helpful. >>>> >>>> On Windows, before this loop in bfs_schedule(): >>>> >>>> + ???????COND_RESET(tstate->bfs_cond); >>>> + ???????while (bfs_thread != tstate) { >>>> + ???????????_bfs_timed_wait(tstate, timestamp); >>>> + ???????????timestamp = get_timestamp(); >>>> + ???????} >>>> >>>> You might want to avoid the call to reset the condition variable if the >>>> predicate is already False. >>>> >>>> -peter >>>> >>>> >>>> >>>> On 4/12/10 8:12 AM, "Nir Aides" >>>> > wrote: >>>> >>>>> Hi Peter, >>>>> >>>>> There is no need for a loop in bfs_yield().? >>>>> >>>>> >>>>> On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante >>>>> >>>>> > wrote: >>>>>> Nir, >>>>>> >>>>>> Per the POSIX standard, both pthread_cond_wait() and >>>>>> pthread_cond_timedwait() need to be performed in a loop. ?See the fourth >>>>>> paragraph of the description from: >>>>>> >>>>>>> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_tim >>>>>>> edwait.html >>>>>>> >>>>>> medwait.html> >>>>>> >>>>>> For the Windows side, I think you have a similar problem. Condition >>>>>> variables are signaling mechanisms, and so they have a separate boolean >>>>>> predicate associated with them. If you release the mutex that protects >>>>>> the predicate, then after you reacquire the mutex, you have to reevaluate >>>>>> the predicate to ensure that the condition has actually been met. >>>>>> >>>>>> You might want to look at the following for a discussion (not sure how >>>>>> good it is, as I just google?d it quickly) of how to implement POSIX >>>>>> semantics on Windows: >>>>>> >>>>>>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>>>> >>>>>> >>>>>> Before you can evaluate the effectiveness of any of the proposed >>>>>> scheduling schemes, the fundamental uses of mutexes and condition >>>>>> variables, and their implementations, must be sound. >>>>>> >>>>>> -peter >>>>>> >>>>>> >>>>>> >>>>>> On 4/11/10 6:50 PM, "Nir Aides" < >>>>>> >>>>>> nir at winpdb.org >>>>>> > wrote: >>>>>> >>>>>>> Hello all, >>>>>>> >>>>>>> I would like to kick this discussion back to life with a simplified >>>>>>> implementation of the BFS scheduler, designed by the Linux kernel hacker >>>>>>> Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt >>>>>>> >>>>>>> >>>>>>> I submitted bfs.patch at? http://bugs.python.org/issue7946 >>>>>>> . It is work in progress but is ready >>>>>>> for some opinion. >>>>>>> >>>>>>> >>>>>>> On my machine BFS gives comparable performance to gilinter, and seems to >>>>>>> schedule threads more fairly, predictably, and with lower rate of >>>>>>> context switching.?Its basic design is very simple but nevertheless it >>>>>>> was designed by an expert in this field, two?characteristics?which >>>>>>> combine to make it attractive to this case. >>>>>>> >>>>>>> The problem addressed by the GIL has always been *scheduling* threads to >>>>>>> the interpreter, not just controlling access to it, and therefore the >>>>>>> GIL, a lock implemented as a simple semaphore was the wrong solution. >>>>>>> >>>>>>> The patches by Antoine and David essentially evolve the GIL into a >>>>>>> scheduler, however both cause thread starvation or high rate of context >>>>>>> switching in some scenarios: >>>>>>> >>>>>>> With Floren't write test ( http://bugs.python.org/issue7946#msg101120 >>>>>>> ): >>>>>>> >>>>>>> 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context >>>>>>> switching shoots up to 200K/s. >>>>>>> 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter >>>>>>> patch starves one of the bg threads. >>>>>>> 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads >>>>>>> starved, context switching shoots up to 250K/s. >>>>>>> >>>>>>> With UDP test ( http://bugs.python.org/file16316/udp-iotest.py >>>>>>> ), add >>>>>>> zlib.compress(b'GIL') to the workload: >>>>>>> >>>>>>> both gilinter and PyCon patches starve the IO thread. >>>>>>> >>>>>>> The BFS patch currently involves more overhead by reading the time stamp >>>>>>> on each yield and schedule operations. In addition it still remains to >>>>>>> address some issues related to timestamps such as getting different time >>>>>>> stamp readings on different cores on some (older) multi-core systems. >>>>>>> >>>>>>> Any thoughts? >>>>>>> >>>>>>> Nir >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < >>>>>>> >>>>>>> solipsis at pitrou.net >>>>>>> > wrote: Hello, As some of you may know, Dave Beazley recently exhibited a situation where the new GIL shows quite a poor behaviour (the old GIL isn't very good either, but still a little better). This issue is followed in http://bugs.python.org/issue7946 This situation is when an IO-bound thread wants to process a lot of incoming packets, while one (or several) CPU-bound thread is also running. Each time the IO-bound thread releases the GIL, the CPU-bound thread gets it and keeps holding it for at least 5 milliseconds (default setting), which limits the number of individual packets which can be recv()'ed and processed per second. I have proposed two mechanisms, based on the same idea: IO-bound threads should be able to steal the GIL very quickly, rather than having to wait for the whole "thread switching interval" (again, 5 ms by default). They differ in how they detect an "IO-bound threads": - the first mechanism is actually the same mechanism which was ??embodied in the original new GIL patch before being removed. In this ??approach, IO methods (such as socket.read() in socketmodule.c) ??releasing the GIL must use a separate C macro when trying to get the ??GIL back again. - the second mechanism dynamically computes the "interactiveness" of a ??thread and allows interactive threads to steal the GIL quickly. In ??this approach, IO methods don't have to be modified at all. Both approaches show similar benchmark results (for the benchmarks that I know of) and basically fix the issue put forward by Dave Beazley. Any thoughts? Regards Antoine. ______________________________ _________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Python-Dev mailing list >>>>>>> Python-Dev at python.org >>>>>>> >>>>>>> http://mail.python.org/mailman/listinfo/python-dev >>>>>>> >>>>>>> Unsubscribe: >>>>>>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gma >>>>>>> il.com >>>>>>> >>>>>> ail.com> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Python-Dev mailing list >>>>> Python-Dev at python.org >>>>> >>>>> http://mail.python.org/mailman/listinfo/python-dev >>>>> Unsubscribe: >>>>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail >>>>> .com >>> >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> http://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.c >>> om > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gzlist at googlemail.com Mon Apr 12 22:27:50 2010 From: gzlist at googlemail.com (Martin (gzlist)) Date: Mon, 12 Apr 2010 21:27:50 +0100 Subject: [Python-Dev] Issue #7978, unexpected EINTR-related exceptions In-Reply-To: <201004081111.16299.victor.stinner@haypocalc.com> References: <201004081111.16299.victor.stinner@haypocalc.com> Message-ID: On 08/04/2010, Victor Stinner wrote: > Le jeudi 08 avril 2010 08:11:09, Yaniv Aknin a ?crit : >> Issue #7978 (http://bugs.python.org/issue7978) describes a bug in >> SocketServer where a received signal in SocketServer's select() call >> will raise an uncaught exception due to EINTR. The proposed solution >> was to wrap SocketServer's select() with something like twisted's >> untilConcludes function, which catches EINTR exceptions and re-calls >> the call (see issue for code). >> >> However, before committing this to SocketServer, neologix raised the >> (valid) concern that this is generally-useful code, already duplicated >> at least in subprocess (_eintr_retry_call, subprocess.py), which can >> probably be moved to another module and be shared from there. (...) > > +1 to share EINTR-related code, but I don't know the best place for such > functions. Maybe the os module? Bazaar did a similar thing in copying the Twisted function to a generic module for use throughout the codebase. From the experience there, I think replicating that would be a bad idea. There are a very large number of functions in the Python standard library that can throw EINTR, but hardly any of them can be "fixed" by wrapping them in a function that catches the error and mindlessly recalls the function with the same arguments. In Bazaar, until_no_eintr proved to be an attractive nuisance that lead to incorrect code. Providing such an easily misused interface in the standard library is likely to do more harm than good. There are some errors and other confusions in the initial post and later, but reading this thread from the Bazaar list should give an impression: Martin From jcea at jcea.es Mon Apr 12 23:51:22 2010 From: jcea at jcea.es (Jesus Cea) Date: Mon, 12 Apr 2010 23:51:22 +0200 Subject: [Python-Dev] stdlib socket usage and "keepalive" Message-ID: <4BC395DA.3030702@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Debugging a strange problem today, I got the following result: Sockets open by stdlib libraries are open without the "keepalive" option, so the system default is used. The system default under linux is "no keepalive". So, if you are using a URLlib connection, POP3 connection, IMAP connection, etc., any stdlib that internally creates a socket, and your server goes away suddendly (you lose network connectivity, by instance), the library will wait FOREVER for the server. The client can't detect that the server is not longer available. The "keepalive" option will send a probe packed every X minutes of inactivity, to check if the other side is still alive, even if the connection is idle. The issue is bad, but the solution is simple enough. Options: 1. All "client" libraries should create sockets with the "KEEPALIVE" option. 2. Modify the socket C module to create all sockets as "Keepalive" by default. 3. To have a global variable in the socket module to change the default for future sockets. Something like current "socket.setdefaulttimeout()". The default should be "keepalive". 4. Modify client libraries to accept a new optional socket-like object as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example). This is an issue in Linux because by default the sockets are not "keepalive". In other Unix systems, the default is "keepalive". I don't know about MS Windows. What do you think?. The solution seems trivial, after deciding the right way to go. PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS8OV2Zlgi5GaxT1NAQIJhwP8CH+gij4KfrJ1oHW5ys4PboH5Ru0pgdly Wbsza0+uj3p68P1vDnC9jIr7j+fI3ql3DOc8zUoIKGpJoaWVspbv3c4vI4ATLHo+ J6I18dpkviRT8/sT/69vMvghaGndiO0Sks/S4tDjhNstYH7oGjWxi63cKqtGPY/p WSTLpwrd4SY= =vkT4 -----END PGP SIGNATURE----- From guido at python.org Tue Apr 13 00:09:43 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Apr 2010 15:09:43 -0700 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: <4BC395DA.3030702@jcea.es> References: <4BC395DA.3030702@jcea.es> Message-ID: Are you sure about this? ISTM that in most cases when a server goes away unexpectedly the local host will discover this when it next tries to use the socket. Also I recall reading that keepalives are a very controversial concept (since they may actually break connections unnecessarily if the internet merely has a hiccup). --Guido On Mon, Apr 12, 2010 at 2:51 PM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Debugging a strange problem today, I got the following result: > > Sockets open by stdlib libraries are open without the "keepalive" > option, so the system default is used. The system default under linux is > "no keepalive". > > So, if you are using a URLlib connection, POP3 connection, IMAP > connection, etc., any stdlib that internally creates a socket, and your > server goes away suddendly (you lose network connectivity, by instance), > the library will wait FOREVER for the server. The client can't detect > that the server is not longer available. > > The "keepalive" option will send a probe packed every X minutes of > inactivity, to check if the other side is still alive, even if the > connection is idle. > > The issue is bad, but the solution is simple enough. Options: > > 1. All "client" libraries should create sockets with the "KEEPALIVE" option. > > 2. Modify the socket C module to create all sockets as "Keepalive" by > default. > > 3. To have a global variable in the socket module to change the default > for future sockets. Something like current "socket.setdefaulttimeout()". > The default should be "keepalive". > > 4. Modify client libraries to accept a new optional socket-like object > as an optional parameter. This would allow things like transparent > compression or encryption, or to replace the socket connection by > anything else (read/write to shared memory or database, for example). > > This is an issue in Linux because by default the sockets are not > "keepalive". In other Unix systems, the default is "keepalive". I don't > know about MS Windows. > > What do you think?. The solution seems trivial, after deciding the right > way to go. > > PS: "socket.setdefaulttimeout()" is not enough, because it could > shutdown a perfectly functional connection, just because it was idle for > too long. > > - -- > Jesus Cea Avion ? ? ? ? ? ? ? ? ? ? ? ? _/_/ ? ? ?_/_/_/ ? ? ? ?_/_/_/ > jcea at jcea.es - http://www.jcea.es/ ? ? _/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > jabber / xmpp:jcea at jabber.org ? ? ? ? _/_/ ? ?_/_/ ? ? ? ? ?_/_/_/_/_/ > . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?_/_/ ?_/_/ ? ?_/_/ ? ? ? ? ?_/_/ ?_/_/ > "Things are not so easy" ? ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > "My name is Dump, Core Dump" ? _/_/_/ ? ? ? ?_/_/_/ ? ? ?_/_/ ?_/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQCVAwUBS8OV2Zlgi5GaxT1NAQIJhwP8CH+gij4KfrJ1oHW5ys4PboH5Ru0pgdly > Wbsza0+uj3p68P1vDnC9jIr7j+fI3ql3DOc8zUoIKGpJoaWVspbv3c4vI4ATLHo+ > J6I18dpkviRT8/sT/69vMvghaGndiO0Sks/S4tDjhNstYH7oGjWxi63cKqtGPY/p > WSTLpwrd4SY= > =vkT4 > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From jcea at jcea.es Tue Apr 13 00:34:27 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 13 Apr 2010 00:34:27 +0200 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: References: <4BC395DA.3030702@jcea.es> Message-ID: <4BC39FF3.6020704@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/13/2010 12:09 AM, Guido van Rossum wrote: > Are you sure about this? ISTM that in most cases when a server goes > away unexpectedly the local host will discover this when it next tries > to use the socket. Also I recall reading that keepalives are a very > controversial concept (since they may actually break connections > unnecessarily if the internet merely has a hiccup). The case is this: 1. The client does a request. Wait "read"ing the answer. 2. The request is "slow", so the server "acks" the TCP datagram and start to process the request. 3. Now the connection is idle. The server is working and the client is waiting, blocked in the socket "read()". 4. Now you switch off the server (or unplug the ethernet wire). 5. If the client kernel is not doing "keepalive", the client will be blocked FOREVER (days), waiting for the reply. If the client uses "keepalive", its kernel will send a probe after a while (30 minutes, for instance), the probe will fail, and the kernel will shutdown the connection. The problem is: linux doesn't uses KEEPALIVE by default. I have validated this behaviour with Ubuntu 9.10 and a network sniffer. About controversial... keepalive are usually sent only when the connection is 100% idle for a while, when "while" can be >15 minutes, so the load should be "none" for regular connections. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS8Of85lgi5GaxT1NAQJaKwP+P9WvbWqmRyHRvNJFB2wLj87EanyOIN1e TP646wUHJQtuU3XCyAWF53uM4rDGsbVh9j4TGK7+C1SHoRpPHlLMUdfwddtk8nK3 Owmo0V10sfHrFi1E0D5Ub/LXd1GG0ec7Q7OGN30nUR//hCuLe57ckEodwNQGzmtA As5yJ5iwGFw= =GOP1 -----END PGP SIGNATURE----- From jcea at jcea.es Tue Apr 13 00:41:40 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 13 Apr 2010 00:41:40 +0200 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: References: <4BC395DA.3030702@jcea.es> Message-ID: <4BC3A1A4.9040105@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/13/2010 12:09 AM, Guido van Rossum wrote: > Also I recall reading that keepalives are a very > controversial concept (since they may actually break connections > unnecessarily if the internet merely has a hiccup). That is true, but parameters are usually very conservative. In standard Ubuntu box I host out there, the (default) parameters used (if I activate keepalive): + Send the keepalive only after 1800 seconds of complete connection inactivity. + After starting sending keepalives, send NINE probes. + Between probes, wait at least 75 seconds. So you have to have 30 minutes of idle and then at least 11 minutes of no conectivity to "lose" the connection. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS8OhpJlgi5GaxT1NAQIb9AP7BLUAFzsltPr1omW+4+Ox7gF/lLDqNR5V DXejCPJ2oBZdyebcwCSS1djh0thks8nRzG7oq61eA4c+Ax5mvsvW0cY5+BfNcrds j6HwVJK+zgTX6NiO7VdGEysxfYHLbJcJ7PfoOHRWhYzolA7VSJriy8kfvgretTZQ qJreRMaPai8= =MQPh -----END PGP SIGNATURE----- From nir at winpdb.org Tue Apr 13 00:43:27 2010 From: nir at winpdb.org (Nir Aides) Date: Tue, 13 Apr 2010 01:43:27 +0300 Subject: [Python-Dev] "Fixing" the new GIL In-Reply-To: References: Message-ID: There is no need for the "forced" switch to be guaranteed. The motivation for the wait is to increase throughput and decrease latency when OS schedules next thread to run on same core as yielding thread but yielding thread is about to continue running into CPU intensive library code. An occasional spurious wakeup resulting in missed switch will not affect that. Note similar logic in new GIL. I will add the loop to keep the code clean. Thanks, Nir On Mon, Apr 12, 2010 at 10:49 PM, Peter Portante wrote: > That code will not switch to another thread if the condition variable > either does not actually block the thread on the call (which is allowed by > the standard to give implementations some flexibility for making it work > correctly ? read the standard reasoning for more information), or the thread > is woken up without a predicate change (also allowed by the standard for > similar reasons). Both of those cases are called ?spurious wake-ups?. > > You may not be able to readily get your implementation to behavior that > way, but in the wild, we need to account for this behavior because Cpython > will be run on systems where it will happen. :) > > -peter > > > > On 4/12/10 3:36 PM, "Nir Aides" wrote: > > Please describe clearly a step by step scenario in which that code will > fail. > > > On Mon, Apr 12, 2010 at 10:25 PM, Peter Portante < > peter.a.portante at gmail.com> wrote: > > Yes, but unless you loop until the predicate is False, no forced switch is > guaranteed to occur. You might as well remove the code. If you want to keep > the code as is, call me when you need a life guard to help debug mystifying > behaviors. ;) -peter > > > > On 4/12/10 3:17 PM, "Nir Aides" > > wrote: > > The loop-less wait is similar to the one in new GIL. It is used to force a > switch to next thread in particular scenario and the motivation is explained > in comment to another if clause a few lines up. Those two if clauses can be > joined though. > > > On Mon, Apr 12, 2010 at 3:37 PM, Peter Portante < > peter.a.portante at gmail.com > wrote: > > Hmm, so I see in > bfs_yield(): > > + if (tstate != NULL && bfs_thread_switch == tstate) { > + COND_RESET(tstate->bfs_cond); > + COND_WAIT(tstate->bfs_cond, bfs_mutex); > + } > > So, to me, the above code says, ?Wait for the condition that tstate is > either NULL, or bfs_thread_switch does not equal tstate?. So the predicate > is: ?(tstate != NULL && bfs_thread_switch == tstate)?. > > If the predicate is True before you call COND_WAIT() and True after you > call COND_WAIT(), either you don?t need to call COND_WAIT() at all, or you > need to loop until the predicate is False. There is no guarantee that a > condition wait actually did anything at all. Yes, there will be spurious > wake ups, but you can?t tell if the thread actually blocked and then woke > up, or never blocked at all. If it never actually blocks, then that code > path is not helpful. > > On Windows, before this loop in bfs_schedule(): > > + COND_RESET(tstate->bfs_cond); > + while (bfs_thread != tstate) { > + _bfs_timed_wait(tstate, timestamp); > + timestamp = get_timestamp(); > + } > > You might want to avoid the call to reset the condition variable if the > predicate is already False. > > -peter > > > > On 4/12/10 8:12 AM, "Nir Aides" < > http://nir at winpdb.org> > wrote: > > Hi Peter, > > There is no need for a loop in bfs_yield(). > > > On Mon, Apr 12, 2010 at 4:26 AM, Peter Portante < > peter.a.portante at gmail.com < > http://peter.a.portante at gmail.com> > wrote: > > Nir, > > Per the POSIX standard, both pthread_cond_wait() and > pthread_cond_timedwait() need to be performed in a loop. See the fourth > paragraph of the description from: > > > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html< > http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html> > > > > For the Windows side, I think you have a similar problem. Condition > variables are signaling mechanisms, and so they have a separate boolean > predicate associated with them. If you release the mutex that protects the > predicate, then after you reacquire the mutex, you have to reevaluate the > predicate to ensure that the condition has actually been met. > > You might want to look at the following for a discussion (not sure how good > it is, as I just google?d it quickly) of how to implement POSIX semantics on > Windows: > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html < > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html> > > > Before you can evaluate the effectiveness of any of the proposed scheduling > schemes, the fundamental uses of mutexes and condition variables, and their > implementations, must be sound. > > -peter > > > > On 4/11/10 6:50 PM, "Nir Aides" < > > nir at winpdb.org < > http://nir at winpdb.org> > wrote: > > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt < > http://ck.kolivas.org/patches/bfs/sched-BFS.txt> > > I submitted bfs.patch at http://bugs.python.org/issue7946 < > http://bugs.python.org/issue7946> . It is work in progress but is ready > for some opinion. > > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed > by an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it, and therefore the GIL, a > lock implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a > scheduler, however both cause thread starvation or high rate of context > switching in some scenarios: > > With Floren't write test ( http://bugs.python.org/issue7946#msg101120 < > http://bugs.python.org/issue7946#msg101120> ): > > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter > patch starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test ( http://bugs.python.org/file16316/udp-iotest.py < > http://bugs.python.org/file16316/udp-iotest.py> ), add > zlib.compress(b'GIL') to the workload: > > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou < > > solipsis at pitrou.net < > http://solipsis at pitrou.net> > wrote: > > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ______________________________ > _________________ > Python-Dev mailing list > Python-Dev at python.org < > http://Python-Dev at python.org> > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org < > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org> > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org < > http://Python-Dev at python.org> > http://mail.python.org/mailman/listinfo/python-dev < > http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com< > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com> > > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org < > http://Python-Dev at python.org> > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > > > ------------------------------ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Apr 13 00:47:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Apr 2010 22:47:55 +0000 (UTC) Subject: [Python-Dev] stdlib socket usage and "keepalive" References: <4BC395DA.3030702@jcea.es> Message-ID: Jesus Cea jcea.es> writes: > > PS: "socket.setdefaulttimeout()" is not enough, because it could > shutdown a perfectly functional connection, just because it was idle for > too long. The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you). > 1. All "client" libraries should create sockets with the "KEEPALIVE" option. > > 2. Modify the socket C module to create all sockets as "Keepalive" by > default. I don't know whether there are any negative implications of such solutions. Granted, the first one is less radical than the second (because servers wouldn't be impacted). > 4. Modify client libraries to accept a new optional socket-like object > as an optional parameter. This would allow things like transparent > compression or encryption, or to replace the socket connection by > anything else (read/write to shared memory or database, for example). This could be useful too. Regards Antoine. From daniel at stutzbachenterprises.com Tue Apr 13 00:59:28 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Mon, 12 Apr 2010 17:59:28 -0500 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: <4BC39FF3.6020704@jcea.es> References: <4BC395DA.3030702@jcea.es> <4BC39FF3.6020704@jcea.es> Message-ID: On Mon, Apr 12, 2010 at 5:34 PM, Jesus Cea wrote: > The problem is: linux doesn't uses KEEPALIVE by default. > If you believe the problem is with the Linux kernel, perhaps you should take up your case on a more appropriate mailing list? Python's socket module is a fairly low-level module, as it's just a thin wrapper around the corresponding operating system calls. Anyone using it has to be prepared to deal with a certain amount of exposed operating system details. If you want to use TCP KEEPALIVE on Linux, then just call: my_socket_object.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) Most non-trivial applications use select() or poll() to avoid blocking calls and do their own timeout-checking at the application layer, so they don't need KEEPALIVE. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcea at jcea.es Tue Apr 13 01:08:10 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 13 Apr 2010 01:08:10 +0200 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: References: <4BC395DA.3030702@jcea.es> <4BC39FF3.6020704@jcea.es> Message-ID: <4BC3A7DA.7000701@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/13/2010 12:59 AM, Daniel Stutzbach wrote: > Most non-trivial applications use select() or poll() to avoid blocking > calls and do their own timeout-checking at the application layer, so > they don't need KEEPALIVE. I am thinking about python stdlibs like imaplib, poplib, smtplib, ftplib, urllib, etc., etc., ... - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS8On2Zlgi5GaxT1NAQJDhQQAkveJ2pF9wZGSmQTCW1HFpPX4MOtaEjxs AFCLX3WcFf2qwpN82hLluJRfkmIWZDHhWU4bKJ/GKRkGvsWb3zcOHaX0solpibqK yS/gVUsrgd1GuqxyQqhtd4J5+fPwZr5RQ5JrO/PjpLH8CgTq6azjufixO4Cve4Jh X4LO3GMekj8= =ws3T -----END PGP SIGNATURE----- From jcea at jcea.es Tue Apr 13 01:19:40 2010 From: jcea at jcea.es (Jesus Cea) Date: Tue, 13 Apr 2010 01:19:40 +0200 Subject: [Python-Dev] Getting an optional parameter instead of creating a socket internally (was: Re: stdlib socket usage and " keepalive" ) In-Reply-To: References: <4BC395DA.3030702@jcea.es> Message-ID: <4BC3AA8C.70109@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/13/2010 12:47 AM, Antoine Pitrou wrote: > Jesus Cea jcea.es> writes: >> >> PS: "socket.setdefaulttimeout()" is not enough, because it could >> shutdown a perfectly functional connection, just because it was idle for >> too long. > > The socket timeout doesn't shutdown anything. It just puts a limit on how much > time recv() and send() can block. Then it's up to you to detect whether the > server is still alive (for example by pinging it through whatever means the > application protocol gives you). A regular standard library (let say, poplib) would abort, after getting the timeout exception. >> 4. Modify client libraries to accept a new optional socket-like object >> as an optional parameter. This would allow things like transparent >> compression or encryption, or to replace the socket connection by >> anything else (read/write to shared memory or database, for example). > > This could be useful too. I have been thinking about this for years. Do you actually think this could be formally proposed?. What bugs me is that the socket creation is deep inside the stdlibs. You can not control it easily (I have overloaded socket.socket() in the past for controlling the number of concurrent connections to servers, for instance, or providing encryption), and it is difficult to test. If these stdlib methods could accept an optional parameter instead of creating the socket internally, test is trivial, and you can reuse the lib to access the service via an arbitrary object (this weekend I just tunneled TCP/IP via DNS requests/answers, shame on you, airport wifi hotspots!). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS8Oqi5lgi5GaxT1NAQKPpgP/S2V3umt4SpOw+Epxtj3/oxEFcQuh3s1z WNvS59+qY6IK0+/MCvxDiAYliGbj8PY76lXmRodJOzwVFe7uPzLhq4h3gBBv0zXs KvBHdyL5fnp4UEId89L7+SqejgAfpJk3GXYwAnpLyF5iQzaiYzp0rNDOuSBCeAmW jFjzjMMZznQ= =cFb/ -----END PGP SIGNATURE----- From guido at python.org Tue Apr 13 01:23:10 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Apr 2010 16:23:10 -0700 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: References: <4BC395DA.3030702@jcea.es> <4BC39FF3.6020704@jcea.es> Message-ID: On Mon, Apr 12, 2010 at 3:59 PM, Daniel Stutzbach wrote: > On Mon, Apr 12, 2010 at 5:34 PM, Jesus Cea wrote: >> >> The problem is: linux doesn't uses KEEPALIVE by default. > > If you believe the problem is with the Linux kernel, perhaps you should take > up your case on a more appropriate mailing list? > > Python's socket module is a fairly low-level module, as it's just a thin > wrapper around the corresponding operating system calls.? Anyone using it > has to be prepared to deal with a certain amount of exposed operating system > details. Bingo. I expect that changing this will have too many unanticipated ramifications to be safe. > If you want to use TCP KEEPALIVE on Linux, then just call: > ??? my_socket_object.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) > > Most non-trivial applications use select() or poll() to avoid blocking calls > and do their own timeout-checking at the application layer, so they don't > need KEEPALIVE. > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Apr 13 01:24:34 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Apr 2010 16:24:34 -0700 Subject: [Python-Dev] Getting an optional parameter instead of creating a socket internally (was: Re: stdlib socket usage and " keepalive" ) In-Reply-To: <4BC3AA8C.70109@jcea.es> References: <4BC395DA.3030702@jcea.es> <4BC3AA8C.70109@jcea.es> Message-ID: On Mon, Apr 12, 2010 at 4:19 PM, Jesus Cea wrote: > On 04/13/2010 12:47 AM, Antoine Pitrou wrote: >> Jesus Cea jcea.es> writes: >>> >>> PS: "socket.setdefaulttimeout()" is not enough, because it could >>> shutdown a perfectly functional connection, just because it was idle for >>> too long. >> >> The socket timeout doesn't shutdown anything. It just puts a limit on how much >> time recv() and send() can block. Then it's up to you to detect whether the >> server is still alive (for example by pinging it through whatever means the >> application protocol gives you). > > A regular standard library (let say, poplib) would abort, after getting > the timeout exception. > >>> 4. Modify client libraries to accept a new optional socket-like object >>> as an optional parameter. This would allow things like transparent >>> compression or encryption, or to replace the socket connection by >>> anything else (read/write to shared memory or database, for example). >> >> This could be useful too. > > I have been thinking about this for years. Do you actually think this > could be formally proposed?. > > What bugs me is that the socket creation is deep inside the stdlibs. You > can not control it easily (I have overloaded socket.socket() in the past > for controlling the number of concurrent connections to servers, for > instance, or providing encryption), and it is difficult to test. > > If these stdlib methods could accept an optional parameter instead of > creating the socket internally, test is trivial, and you can reuse the > lib to access the service via an arbitrary object (this weekend I just > tunneled TCP/IP via DNS requests/answers, shame on you, airport wifi > hotspots!). Yeah, this sounds like a useful change. Would be very useful for testing too. -- --Guido van Rossum (python.org/~guido) From exarkun at twistedmatrix.com Tue Apr 13 04:03:19 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 13 Apr 2010 02:03:19 -0000 Subject: [Python-Dev] Getting an optional parameter instead of creating a socket internally (was: Re: stdlib socket usage and " keepalive" ) In-Reply-To: <4BC3AA8C.70109@jcea.es> References: <4BC395DA.3030702@jcea.es> <4BC3AA8C.70109@jcea.es> Message-ID: <20100413020319.2779.2069488812.divmod.xquotient.401@localhost.localdomain> On 12 Apr, 11:19 pm, jcea at jcea.es wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >On 04/13/2010 12:47 AM, Antoine Pitrou wrote: >>Jesus Cea jcea.es> writes: >>> >>>PS: "socket.setdefaulttimeout()" is not enough, because it could >>>shutdown a perfectly functional connection, just because it was idle >>>for >>>too long. >> >>The socket timeout doesn't shutdown anything. It just puts a limit on >>how much >>time recv() and send() can block. Then it's up to you to detect >>whether the >>server is still alive (for example by pinging it through whatever >>means the >>application protocol gives you). > >A regular standard library (let say, poplib) would abort, after getting >the timeout exception. >>>4. Modify client libraries to accept a new optional socket-like >>>object >>>as an optional parameter. This would allow things like transparent >>>compression or encryption, or to replace the socket connection by >>>anything else (read/write to shared memory or database, for example). >> >>This could be useful too. > >I have been thinking about this for years. Do you actually think this >could be formally proposed?. Every once in a while I make a little bit more progress on the PEP I'm working on for this. If you want to talk more about this, you can find me in #python-dev or #twisted on freenode. Jean-Paul From greg.ewing at canterbury.ac.nz Tue Apr 13 09:35:09 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Apr 2010 19:35:09 +1200 Subject: [Python-Dev] Traceback object has no __class__? In-Reply-To: <4BC2C412.2060306@gmail.com> References: <4BC1A7C2.4090003@canterbury.ac.nz> <4BC2C412.2060306@gmail.com> Message-ID: <4BC41EAD.9030704@canterbury.ac.nz> Nick Coghlan wrote: > I'm not sure what build you're getting that behaviour on, but my svn > build of 2.6 has a __class__ attribute for traceback objects, It's 2.6.1. Guess it's been fixed since then. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 13 10:43:58 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Apr 2010 20:43:58 +1200 Subject: [Python-Dev] stdlib socket usage and "keepalive" In-Reply-To: <4BC39FF3.6020704@jcea.es> References: <4BC395DA.3030702@jcea.es> <4BC39FF3.6020704@jcea.es> Message-ID: <4BC42ECE.8000309@canterbury.ac.nz> Jesus Cea wrote: > About controversial... keepalive are usually sent only when the > connection is 100% idle for a while, when "while" can be >15 minutes, so > the load should be "none" for regular connections. I guess the concern would be that the keepalive probe itself is subject to uncertain delays, so how long do you wait for a reply before deciding that the connection is dead? If you don't wait long enough, you could end up killing a connection that would have come back if you had waited longer. -- Greg From ncoghlan at gmail.com Tue Apr 13 14:54:46 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Apr 2010 22:54:46 +1000 Subject: [Python-Dev] OS information, tags In-Reply-To: <20100412135151.GA31685@remy> References: <20100412135151.GA31685@remy> Message-ID: <4BC46996.4020909@gmail.com> Senthil Kumaran wrote: > On Mon, Apr 12, 2010 at 07:06:29PM +0530, Anand Balachandran Pillai wrote: >> I am surprised to see that the bug-tracker >> doesn't have an OS classifier or ability to add >> tags ? Since a number of issues reported seem to > > There is one. In the Components you can do a multiple select and it > has Macintosh , Windows as options. I think that setup dates from the Sourceforge days when we didn't have keywords or the ability to add our own fields. Would it make sense to put a request on the metatracker to convert these to keywords now that they're available? Or even a separate OS field with "Windows, Mac OS X, Linux, *BSD, Other" as the options? While there is some Windows and Mac specific code, treating them as separate components seems fairly unintuitive. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Apr 13 15:04:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Apr 2010 23:04:18 +1000 Subject: [Python-Dev] Getting an optional parameter instead of creating a socket internally In-Reply-To: <4BC3AA8C.70109@jcea.es> References: <4BC395DA.3030702@jcea.es> <4BC3AA8C.70109@jcea.es> Message-ID: <4BC46BD2.2080403@gmail.com> Jesus Cea wrote: > On 04/13/2010 12:47 AM, Antoine Pitrou wrote: >> Jesus Cea jcea.es> writes: >>> 4. Modify client libraries to accept a new optional socket-like object >>> as an optional parameter. This would allow things like transparent >>> compression or encryption, or to replace the socket connection by >>> anything else (read/write to shared memory or database, for example). >> This could be useful too. > > I have been thinking about this for years. Do you actually think this > could be formally proposed?. This strikes me as the best way forward (albeit far from easy to do): 1. Existing code which doesn't use the new arguments should be unaffected. 2. No changes to low level socket code, so no chance of unintended side effects on library behaviour 3. Solves a broad class of problems (i.e. allows all sorts of tunnelling and various socket options) rather than just this one problem. 4. Allows easier library testing with "mock" socket objects. The problem actually reminds me of Spolsky's Law of Leaky Abstractions essay. If the abstraction layer is going to leak anyway (and they always do), it's best to give the developer access to all of the tools they need to plug the hole. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Apr 13 15:13:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Apr 2010 23:13:39 +1000 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <20100413021740.8C16FEE984@mail.python.org> References: <20100413021740.8C16FEE984@mail.python.org> Message-ID: <4BC46E03.6020601@gmail.com> barry.warsaw wrote: > +It is recommended that when nothing sensible can be calculated, > +implementations should set the `__cached__` attribute to `None`. What (if anything) should we set __cached__ to in __main__? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Apr 13 16:04:17 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Apr 2010 10:04:17 -0400 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <4BC46E03.6020601@gmail.com> References: <20100413021740.8C16FEE984@mail.python.org> <4BC46E03.6020601@gmail.com> Message-ID: <20100413100417.2bf4bbbc@heresy> On Apr 13, 2010, at 11:13 PM, Nick Coghlan wrote: >barry.warsaw wrote: >> +It is recommended that when nothing sensible can be calculated, >> +implementations should set the `__cached__` attribute to `None`. > >What (if anything) should we set __cached__ to in __main__? Good catch. Right now (in my current branch) it is undefined. It should be None. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From stephen at xemacs.org Tue Apr 13 16:04:26 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 13 Apr 2010 23:04:26 +0900 Subject: [Python-Dev] OS information, tags In-Reply-To: <4BC46996.4020909@gmail.com> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: <87zl1730ut.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > they're available? Or even a separate OS field with "Windows, Mac OS X, > Linux, *BSD, Other" as the options? XEmacs has a multilink field "platform". The default is "Any or all" (mislabeled N/A), and values include hardware (currently x86, PPC, other), OS (POSIX, Windows NT), and GUI (Windows, X11, Carbon, TTY[sic]). Multilink means you can have as many as you want, and no attempt is made to enforce "one of each" or anything like that. I don't know if the distinction between OS and GUI is as important for Python. And "platform" is rarely useful for us -- platform bugs tend to be spectacular things like build breakage or crashes, and so get attention from a much wider circle than just the platform specialists. (Especially since the guilty party is usually a nonspecialist who never heard of that kind of dragon before.) So I can't really offer a pile of relevant experience, but it seems like a good idea. > While there is some Windows and Mac specific code, treating them as > separate components seems fairly unintuitive. Not always unintuitive. There are some features only available on a particular platform, then "component" sort of makes sense. OTOH, there are some bugs that are only available on a particular platform even though the code is generic. There "component" isn't very intuitive. In theory, I could see having both the component and the platform fields, but probably that would require too much user education to be worth it. From abpillai at gmail.com Tue Apr 13 16:17:16 2010 From: abpillai at gmail.com (Anand Balachandran Pillai) Date: Tue, 13 Apr 2010 19:47:16 +0530 Subject: [Python-Dev] OS information, tags In-Reply-To: <4BC46996.4020909@gmail.com> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: On Tue, Apr 13, 2010 at 6:24 PM, Nick Coghlan wrote: > Senthil Kumaran wrote: > > On Mon, Apr 12, 2010 at 07:06:29PM +0530, Anand Balachandran Pillai > wrote: > >> I am surprised to see that the bug-tracker > >> doesn't have an OS classifier or ability to add > >> tags ? Since a number of issues reported seem to > > > > There is one. In the Components you can do a multiple select and it > > has Macintosh , Windows as options. > Thanks for pointing that out. I missed it. A problem seems to be that for most bugs, the filer seems to select just one component and leave the rest of it to bug description. When I searched with "Windows" component, the most recent bugs I saw was upto 1 week ago, where as a bug like http://bugs.python.org/issue8384 which is a distutils issue reported on Windows is not present, because the reporter chose to file it under "distutils" component. > > I think that setup dates from the Sourceforge days when we didn't have > keywords or the ability to add our own fields. Would it make sense to > put a request on the metatracker to convert these to keywords now that > they're available? Or even a separate OS field with "Windows, Mac OS X, > Linux, *BSD, Other" as the options? > > While there is some Windows and Mac specific code, treating them as > separate components seems fairly unintuitive. > +1. I don't understand why Windows and Mac should come under "Components". It is not very obvious to the reporters also. Separate question: Who is in charge of bug triaging in pydev ? This is something which I would like to help out with, if help is required :-) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > -- --Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Apr 13 16:17:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Apr 2010 00:17:13 +1000 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <20100413100417.2bf4bbbc@heresy> References: <20100413021740.8C16FEE984@mail.python.org> <4BC46E03.6020601@gmail.com> <20100413100417.2bf4bbbc@heresy> Message-ID: <4BC47CE9.9030500@gmail.com> Barry Warsaw wrote: > On Apr 13, 2010, at 11:13 PM, Nick Coghlan wrote: > >> barry.warsaw wrote: >>> +It is recommended that when nothing sensible can be calculated, >>> +implementations should set the `__cached__` attribute to `None`. >> What (if anything) should we set __cached__ to in __main__? > > Good catch. Right now (in my current branch) it is undefined. It should be > None. Sounds reasonable. I ask because the various functions in runpy will also need to cover setting that value properly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Apr 13 16:21:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Apr 2010 00:21:07 +1000 Subject: [Python-Dev] OS information, tags In-Reply-To: <87zl1730ut.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <87zl1730ut.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BC47DD3.60800@gmail.com> Stephen J. Turnbull wrote: > > While there is some Windows and Mac specific code, treating them as > > separate components seems fairly unintuitive. > > Not always unintuitive. There are some features only available on a > particular platform, then "component" sort of makes sense. OTOH, > there are some bugs that are only available on a particular platform > even though the code is generic. There "component" isn't very > intuitive. In theory, I could see having both the component and the > platform fields, but probably that would require too much user > education to be worth it. Yep, that's where the "fairly" qualification came from. It particularly made sense when we had all those Mac OS 9 specific modules kicking around. This isn't something I feel strongly about - just pointing out to the triage guys that requesting a change *is* an option if they think it would help. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Apr 13 16:38:35 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Apr 2010 10:38:35 -0400 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <4BC47CE9.9030500@gmail.com> References: <20100413021740.8C16FEE984@mail.python.org> <4BC46E03.6020601@gmail.com> <20100413100417.2bf4bbbc@heresy> <4BC47CE9.9030500@gmail.com> Message-ID: <20100413103835.1a4cfc09@heresy> On Apr 14, 2010, at 12:17 AM, Nick Coghlan wrote: >Barry Warsaw wrote: >> On Apr 13, 2010, at 11:13 PM, Nick Coghlan wrote: >> >>> barry.warsaw wrote: >>>> +It is recommended that when nothing sensible can be calculated, >>>> +implementations should set the `__cached__` attribute to `None`. >>> What (if anything) should we set __cached__ to in __main__? >> >> Good catch. Right now (in my current branch) it is undefined. It should be >> None. > >Sounds reasonable. I ask because the various functions in runpy will >also need to cover setting that value properly. Right. I'm looking at runpy now, but am less familiar with this code. Right now I have anything executed with -m as setting __cached__ to None, which seems right in the simple case because the -m doesn't appear to get byte compiled. Is that correct? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From daniel at stutzbachenterprises.com Tue Apr 13 18:12:47 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 13 Apr 2010 11:12:47 -0500 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 1:06 PM, Reid Kleckner wrote: > Looking at dictnotes.txt, I can see that people have experimented with > taking advantage of cache locality. I was wondering what benchmarks > were used to glean these lessons before I write my own. Python > obviously has very particular workloads that need to be modeled > appropriately, such as namespaces and **kwargs dicts. > I don't know what benchmarks were used to write dictnotes.txt, but moving forward I would recommend implementing your changes on trunk (i.e., Python 2.x) and running the Unladen Swallow Benchmarks, which you can get from the link below: http://code.google.com/p/unladen-swallow/wiki/Benchmarks They are macro-benchmarks on real applications. You will probably also want to write some micro-benchmarks of your own so that you can pinpoint any bottlenecks in your code and determine where you are ahead of the current dict implementation and where you are behind. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Apr 13 18:51:02 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 13 Apr 2010 12:51:02 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: <20100413165102.58A011FF34F@kimball.webabinitio.net> On Tue, 13 Apr 2010 19:47:16 +0530, Anand Balachandran Pillai wrote: > On Tue, Apr 13, 2010 at 6:24 PM, Nick Coghlan wrote: > > > Senthil Kumaran wrote: > > > On Mon, Apr 12, 2010 at 07:06:29PM +0530, Anand Balachandran Pillai > > wrote: > > >> I am surprised to see that the bug-tracker > > >> doesn't have an OS classifier or ability to add > > >> tags ? Since a number of issues reported seem to > > > > > > There is one. In the Components you can do a multiple select and it > > > has Macintosh , Windows as options. > > > > Thanks for pointing that out. I missed it. > > A problem seems to be that for most bugs, the filer seems to select > just one component and leave the rest of it to bug description. When > I searched with "Windows" component, the most recent bugs I saw > was upto 1 week ago, where as a bug like http://bugs.python.org/issue8384 > which is a distutils issue reported on Windows is not present, because > the reporter chose to file it under "distutils" component. Well, one generally can't depend on the submitter to get the details right, because they often aren't familiar with our conventions. Triage tries to add those components when appropriate, but sometimes we forget. The more interesting question is, do any developers search on the windows/mac components? My guess would be that we mostly depend on appropriate people getting set as nosy on bugs, by themselves after seeing it on the new bugs list, or by triage. > > I think that setup dates from the Sourceforge days when we didn't have > > keywords or the ability to add our own fields. Would it make sense to > > put a request on the metatracker to convert these to keywords now that > > they're available? Or even a separate OS field with "Windows, Mac OS X, > > Linux, *BSD, Other" as the options? > > > > While there is some Windows and Mac specific code, treating them as > > separate components seems fairly unintuitive. > > > > +1. I don't understand why Windows and Mac should come under > "Components". It is not very obvious to the reporters also. Well, on the flip side, if there were a separate OS list, we'd probably get a lot of submitters selecting their OS when in fact the bug is not OS specific. So I'm not sure the problems would be any smaller with that approach. > Separate question: Who is in charge of bug triaging in pydev ? This is > something which I would like to help out with, if help is required :-) As with just about everything else Python, a small, ad hoc team of volunteers. There's no one "in charge". Anyone with 'developer' privileges on the tracker (which means anyone with commit access, but also non-committers who have been given it so they can do triage) can do triage. There are a batch of us that hang out on #python-dev and also monitor the new bugs and/or the full bugs list, and triage issues as we have time. If you want to help out, come talk to us on #python-dev (on freenode), and start reviewing issues and suggesting appropriate status updates. If you help out, it won't be long before you get tracker 'developer' privs so you can help out more. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Tue Apr 13 18:58:27 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 13 Apr 2010 12:58:27 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: <4BC47DD3.60800@gmail.com> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <87zl1730ut.fsf@uwakimon.sk.tsukuba.ac.jp> <4BC47DD3.60800@gmail.com> Message-ID: <20100413165827.07BD81FF350@kimball.webabinitio.net> On Wed, 14 Apr 2010 00:21:07 +1000, Nick Coghlan wrote: > Stephen J. Turnbull wrote: > > > While there is some Windows and Mac specific code, treating them as > > > separate components seems fairly unintuitive. > > > > Not always unintuitive. There are some features only available on a > > particular platform, then "component" sort of makes sense. OTOH, > > there are some bugs that are only available on a particular platform > > even though the code is generic. There "component" isn't very > > intuitive. In theory, I could see having both the component and the > > platform fields, but probably that would require too much user > > education to be worth it. > > Yep, that's where the "fairly" qualification came from. It particularly > made sense when we had all those Mac OS 9 specific modules kicking around. > > This isn't something I feel strongly about - just pointing out to the > triage guys that requesting a change *is* an option if they think it > would help. It's not really a question of what Triage needs, but what the committers as a whole would find most useful. As I said in another note, I think having a separate OS selection would cause just as many triage issues as the current setup does. So the question is, what would developers and bug searchers/submitters find useful? That said, it's easier to add components and keywords than it is to add a new selection box. I know how to do the former but not (yet) the latter. -- R. David Murray www.bitdance.com From tjreedy at udel.edu Tue Apr 13 20:22:40 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Apr 2010 14:22:40 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: On 4/13/2010 10:17 AM, Anand Balachandran Pillai wrote: > >> I am surprised to see that the bug-tracker > >> doesn't have an OS classifier or ability to add > >> tags ? Since a number of issues reported seem to > > > > There is one. In the Components you can do a multiple select and it > > has Macintosh , Windows as options. > > > Thanks for pointing that out. I missed it. Me to. Are thery really never any *nix specific bugs? There are *nix specific features, so bugs in those features would be. > Separate question: Who is in charge of bug triaging in pydev ? This is > something which I would like to help out with, if help is required :-) It is being done well now, but it is possible that some of those doing so could move up to patch review, or something, if there were more triagers. I started by registering and just making unofficial triage comments like 'behavior matches doc' followed by 'recommend closing as invalid' or 'recommend changing to feature request' or 'recommend changing to doc improvement request' as I thought best. Eventually (after about a year) a committer noticed that he (nearly) always agreed and gave me admin privileges to make the changes directly. David's #python-dev suggestion would speed that process ;-) I might even give it a try sometime when I am in doubt and would otherwise not do anything. One way or the other, give it a try. Terry Jan Reedy From db3l.net at gmail.com Tue Apr 13 21:43:38 2010 From: db3l.net at gmail.com (David Bolen) Date: Tue, 13 Apr 2010 15:43:38 -0400 Subject: [Python-Dev] Status of 2.7b1? References: <4BC09E77.8030209@gmail.com> <4BC0C55A.5060001@gmail.com> Message-ID: Brian Curtin writes: > The tests are run on a native Win32 build as compiled by VS2008. The > functionality is Win32 specific and wouldn't work on Cygwin, so the tests > are skipped there. I believe Cygwin is used for kicking off the tests and > other buildbot stuff, but they don't actually run through Cygwin. Yes, that's correct. Cygwin is on my buildbot just for management convenience. The build slave happens to be started from a Cygwin bash shell (and beneath a Cygwin-based home directory, which is why you see it in various path references in logs), but the buildbot code itself is responsible for executing the python process directly for testing. And python itself is built normally with VS 2008 as a pure win32 build, no Cygwin dependency. I did also verify over the weekend that the failures occur whether the python_d process is started from a Cygwin bash shell or from the normal Windows cmd process, and even if test_os is used directly, without running through rt.bat. The case that ran successfully was not via test_os, but only by interactively replicating the test. I'm back at this point, and to the extent my buildbot is the only place currently replicating the problem, am working with Brian for whatever access or resources might be helpful. -- David From barry at python.org Tue Apr 13 22:21:07 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Apr 2010 16:21:07 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging Message-ID: <20100413162107.171c724c@heresy> I am attaching the latest revision of PEP 3147 to this message, which is also available here: http://www.python.org/dev/peps/pep-3147/ I think the PEP is ready for pronouncement, and the patch is pretty much ready for merging into py3k. The only thing that I can think of that is not implemented yet is this section on PEP 302 loaders: PEP 302 [18]_ defined loaders have a `.get_filename()` method which points to the `__file__` for a module. As part of this PEP, we will extend this API, to include a new method `.get_paths()` which will return a 2-tuple containing the path to the source file and the path to where the matching `pyc` file is (or would be). I'm honestly not sure whether this is still essential, or whether the importlib ABC changes Brett and I talked about at Pycon are still required. I now believe they are at best a minor part of the implementation if so. Maybe Brett can chime in on that. Everything else is implemented, tested, and has undergone four rounds of Rietveld reviews (thanks Antoine, Benjamin, Brett, and Georg!). A fifth patch set has been uploaded and is available here: http://codereview.appspot.com/842043/show This addresses all previous comments, includes some fixes from Brian Curtin for Windows (thanks!) and fixes __main__ and -m support. I'd like to commit this to py3k sooner rather than later so that we can shake out any additional issues that might crop up, without having to continue to maintain my external branches. Guido, what say you? -Barry PEP: 3147 Title: PYC Repository Directories Version: $Revision: 80025 $ Last-Modified: $Date: 2010-04-12 22:17:40 -0400 (Mon, 12 Apr 2010) $ Author: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2009-12-16 Python-Version: 3.2 Post-History: 2010-01-30, 2010-02-25, 2010-03-03, 2010-04-12 Abstract ======== This PEP describes an extension to Python's import mechanism which improves sharing of Python source code files among multiple installed different versions of the Python interpreter. It does this by allowing more than one byte compilation file (.pyc files) to be co-located with the Python source file (.py file). The extension described here can also be used to support different Python compilation caches, such as JIT output that may be produced by an Unladen Swallow [1]_ enabled C Python. Background ========== CPython compiles its source code into "byte code", and for performance reasons, it caches this byte code on the file system whenever the source file has changes. This makes loading of Python modules much faster because the compilation phase can be bypassed. When your source file is `foo.py`, CPython caches the byte code in a `foo.pyc` file right next to the source. Byte code files contain two 32-bit numbers followed by the marshaled [2]_ code object. The 32-bit numbers represent a magic number and a timestamp. The magic number changes whenever Python changes the byte code format, e.g. by adding new byte codes to its virtual machine. This ensures that pyc files built for previous versions of the VM won't cause problems. The timestamp is used to make sure that the pyc file is not older than the py file that was used to create it. When either the magic number or timestamp do not match, the py file is recompiled and a new pyc file is written. In practice, it is well known that pyc files are not compatible across Python major releases. A reading of import.c [3]_ in the Python source code proves that within recent memory, every new CPython major release has bumped the pyc magic number. Rationale ========= Linux distributions such as Ubuntu [4]_ and Debian [5]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. This causes a conflict for Python source files installed by the system (including third party packages), because you cannot compile a single Python source file for more than one Python version at a time. Thus if your system wanted to install a `/usr/share/python/foo.py`, it could not create a `/usr/share/python/foo.pyc` file usable across all installed Python versions. Furthermore, in order to ease the burden on operating system packagers for these distributions, the distribution packages do not contain Python version numbers [6]_; they are shared across all Python versions installed on the system. Putting Python version numbers in the packages would be a maintenance nightmare, since all the packages - *and their dependencies* - would have to be updated every time a new Python release was added or removed from the distribution. Because of the sheer number of packages available, this amount of work is infeasible. C extensions can be source compatible across multiple versions of Python. Compiled extension modules are usually not compatible though, and PEP 384 [7]_ has been proposed to address this by defining a stable ABI for extension modules. Because these distributions cannot share pyc files, elaborate mechanisms have been developed to put the resulting pyc files in non-shared locations while the source code is still shared. Examples include the symlink-based Debian regimes python-support [8]_ and python-central [9]_. These approaches make for much more complicated, fragile, inscrutable, and fragmented policies for delivering Python applications to a wide range of users. Arguably more users get Python from their operating system vendor than from upstream tarballs. Thus, solving this pyc sharing problem for CPython is a high priority for such vendors. This PEP proposes a solution to this problem. Proposal ======== Python's import machinery is extended to write and search for byte code cache files in a single directory inside every Python package directory. This directory will be called `__pycache__`. Further, pyc files will contain a magic string that differentiates the Python version they were compiled for. This allows multiple byte compiled cache files to co-exist for a single Python source file. This scheme has the added benefit of reducing the clutter in a Python package directory. When a Python source file is imported for the first time, a `__pycache__` directory will be created in the package directory, if one does not already exist. The pyc file for the imported source will be written to the `__pycache__` directory, using the magic-tag formatted name. If either the creation of the `__pycache__` directory or the pyc file inside that fails, the import will still succeed, just as it does in a pre-PEP-3147 world. If the py source file is missing, the pyc file inside `__pycache__` will be ignored. This eliminates the problem of accidental stale pyc file imports. For backward compatibility, Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file *would* have been, i.e. not in the `__pycache__` directory. pyc file outside of `__pycache__` will only be imported if the py source file is missing. Tools such as `py_compile` [15]_ and `compileall` [16]_ will be extended to create PEP 3147 formatted layouts automatically, but will have an option to create pyc-only distribution layouts. Examples --------- What would this look like in practice? Let's say we have a Python package named `alpha` which contains a sub-package name `beta`. The source directory layout before byte compilation might look like this:: alpha/ __init__.py one.py two.py beta/ __init__.py three.py four.py After byte compiling this package with Python 3.2, you would see the following layout:: alpha/ __pycache__/ __init__.cpython-32.pyc one.cpython-32.pyc two.cpython-32.pyc __init__.py one.py two.py beta/ __pycache__/ __init__.cpython-32.pyc three.cpython-32.pyc four.cpython-32.pyc __init__.py three.py four.py *Note: listing order may differ depending on the platform.* Let's say that two new versions of Python are installed, one is Python 3.3 and another is Unladen Swallow. After byte compilation, the file system would look like this:: alpha/ __pycache__/ __init__.cpython-32.pyc __init__.cpython-33.pyc __init__.unladen-10.pyc one.cpython-32.pyc one.cpython-33.pyc one.unladen-10.pyc two.cpython-32.pyc two.cpython-33.pyc two.unladen-10.pyc __init__.py one.py two.py beta/ __pycache__/ __init__.cpython-32.pyc __init__.cpython-33.pyc __init__.unladen-10.pyc three.cpython-32.pyc three.cpython-33.pyc three.unladen-10.pyc four.cpython-32.pyc four.cpython-33.pyc four.unladen-10.pyc __init__.py three.py four.py As you can see, as long as the Python version identifier string is unique, any number of pyc files can co-exist. These identifier strings are described in more detail below. A nice property of this layout is that the `__pycache__` directories can generally be ignored, such that a normal directory listing would show something like this:: alpha/ __pycache__/ __init__.py one.py two.py beta/ __pycache__/ __init__.py three.py four.py This is much less cluttered than even today's Python. Python behavior =============== When Python searches for a module to import (say `foo`), it may find one of several situations. As per current Python rules, the term "matching pyc" means that the magic number matches the current interpreter's magic number, and the source file is not newer than the `pyc` file. Case 1: The first import ------------------------ When Python is asked to import module `foo`, it searches for a `foo.py` file (or `foo` package, but that's not important for this discussion) along its `sys.path`. When Python locates the `foo.py` file it will look for a `__pycache__` directory in the directory where it found the `foo.py`. If the `__pycache__` directory is missing, Python will create it. Then it will parse and byte compile the `foo.py` file and save the byte code in `__pycache__/foo..pyc`, where is defined by the Python implementation, but will be a human readable string such as `cpython-32`. Case 2: The second import ------------------------- When Python is asked to import module `foo` a second time (in a different process of course), it will again search for the `foo.py` file along its `sys.path`. When Python locates the `foo.py` file, it looks for a matching `__pycache__/foo..pyc` and finding this, it reads the byte code and continues as usual. Case 3: __pycache__/foo..pyc with no source --------------------------------------------------- It's possible that the `foo.py` file somehow got removed, while leaving the cached pyc file still on the file system. If the `__pycache__/foo..pyc` file exists, but the `foo.py` file used to create it does not, Python will raise an `ImportError` when asked to import foo. In other words, Python will not import a pyc file from the cache directory unless the source file exists. Case 4: legacy pyc files and source-less imports ------------------------------------------------ Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a `foo.pyc` file exists next to the `foo.py` file, the pyc file will be ignored in all cases In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been. Case 5: read-only file systems ------------------------------ When the source lives on a read-only file system, or the `__pycache__` directory or pyc file cannot otherwise be written, all the same rules apply. This is also the case when `__pycache__` happens to be written with permissions which do not allow for writing containing pyc files. Flow chart ========== Here is a flow chart describing how modules are loaded: .. image:: pep-3147-1.png :scale: 75 Magic identifiers ================= pyc files inside of the `__pycache__` directories contain a magic identifier in their file names. These are mnemonic tags for the actual magic numbers used by the importer. For example, in Python 3.2, we could use the hexlified [10]_ magic number as a unique identifier:: >>> from binascii import hexlify >>> from imp import get_magic >>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii')) 'foo.580c0d0a.pyc' This isn't particularly human friendly though. Instead, this PEP proposes a *magic tag* that uniquely defines `.pyc` files for the current version of Python. Whenever the magic number is bumped, a new magic tag is defined which is unique among all versions and implementations of Python. The actual contents of the magic tag is left up to the implementation, although it is recommended that the tag include the implementation name and a version shorthand. In general, magic numbers never change between Python micro releases, but the convention can be extended to handle magic number changes between pre-release development versions. For example, CPython 3.2 would have a magic tag of `cpython-32` and write pyc files like this: `foo.cpython-32.pyc`. When the `-O` flag is used, it would write `foo.cpython-32.pyo`. For backports of this feature to Python 2, when the `-U` flag is used, a file such as `foo.cpython-27u.pyc` can be written. The magic tag is available in the `imp` module via the `get_tag()` function. This is analogous to the `get_magic()` function already available in that module. Alternative Python implementations ================================== Alternative Python implementations such as Jython [11]_, IronPython [12]_, PyPy [13]_, Pynie [14]_, and Unladen Swallow can also use the `__pycache__` directory to store whatever compilation artifacts make sense for their platforms. For example, Jython could store the class file for the module in `__pycache__/foo.jython-32.class`. Implementation strategy ======================= This feature is targeted for Python 3.2, solving the problem for those and all future versions. It may be back-ported to Python 2.7. Vendors are free to backport the changes to earlier distributions as they see fit. Effects on existing code ======================== Adoption of this PEP will affect existing code and idioms, both inside Python and outside. This section enumerates some of these effects. __file__ --------- In Python 3, when you import a module, its `__file__` attribute points to its source `py` file (in Python 2, it points to the `pyc` file). A package's `__file__` points to the `py` file for its `__init__.py`. E.g.:: >>> import foo >>> foo.__file__ 'foo.py' # baz is a package >>> import baz >>> baz.__file__ 'baz/__init__.py' Nothing in this PEP would change the semantics of `__file__`. This PEP proposes the addition of an `__cached__` attribute to modules, which will always point to the actual `pyc` file that was read or written. When the environment variable `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if the source lives on a read-only filesystem, then the `__cached__` attribute will point to the location that the `pyc` file *would* have been written to if it didn't exist. This location of course includes the `__pycache__` subdirectory in its path. For alternative Python implementations which do not support `pyc` files, the `__cached__` attribute may point to whatever information makes sense. E.g. on Jython, this might be the `.class` file for the module: `__pycache__/foo.jython-32.class`. Some implementations may use multiple compiled files to create the module, in which case `__cached__` may be a tuple. The exact contents of `__cached__` are Python implementation specific. It is recommended that when nothing sensible can be calculated, implementations should set the `__cached__` attribute to `None`. py_compile and compileall ------------------------- Python comes with two modules, `py_compile` [15]_ and `compileall` [16]_ which support compiling Python modules external to the built-in import machinery. `py_compile` in particular has intimate knowledge of byte compilation, so these will be updated to understand the new layout. The `-b` flag is added to `compileall` for writing legacy `.pyc` byte-compiled file path names. bdist_wininst and the Windows installer --------------------------------------- These tools also compile modules explicitly on installation. If they do not use `py_compile` and `compileall`, then they would also have to be modified to understand the new layout. File extension checks --------------------- There exists some code which checks for files ending in `.pyc` and simply chops off the last character to find the matching `.py` file. This code will obviously fail once this PEP is implemented. To support this use case, we'll add two new methods to the `imp` package [17]_: * `imp.source_from_cache(py_path)` -> `pyc_path` * `imp.cache_from_source(pyc_path)` -> `py_path` Alternative implementations are free to override these functions to return reasonable values based on their own support for this PEP. PEP 302 loaders --------------- PEP 302 [18]_ defined loaders have a `.get_filename()` method which points to the `__file__` for a module. As part of this PEP, we will extend this API, to include a new method `.get_paths()` which will return a 2-tuple containing the path to the source file and the path to where the matching `pyc` file is (or would be). Backports --------- For versions of Python earlier than 3.2 (and possibly 2.7), it is possible to backport this PEP. However, in Python 3.2 (and possibly 2.7), this behavior will be turned on by default, and in fact, it will replace the old behavior. Backports will need to support the old layout by default. We suggest supporting PEP 3147 through the use of an environment variable called `$PYTHONENABLECACHEDIR` or the command line switch `-Xenablecachedir` to enable the feature. Makefiles and other dependency tools ------------------------------------ Makefiles and other tools which calculate dependencies on `.pyc` files (e.g. to byte-compile the source if the `.pyc` is missing) will have to be updated to check the new paths. Alternatives ============ PEP 304 ------- There is some overlap between the goals of this PEP and PEP 304 [19]_, which has been withdrawn. However PEP 304 would allow a user to create a shadow file system hierarchy in which to store `pyc` files. This concept of a shadow hierarchy for `pyc` files could be used to satisfy the aims of this PEP. Although the PEP 304 does not indicate why it was withdrawn, shadow directories have a number of problems. The location of the shadow `pyc` files would not be easily discovered and would depend on the proper and consistent use of the `$PYTHONBYTECODE` environment variable both by the system and by end users. There are also global implications, meaning that while the system might want to shadow `pyc` files, users might not want to, but the PEP defines only an all-or-nothing approach. As an example of the problem, a common (though fragile) Python idiom for locating data files is to do something like this:: from os import dirname, join import foo.bar data_file = join(dirname(foo.bar.__file__), 'my.dat') This would be problematic since `foo.bar.__file__` will give the location of the `pyc` file in the shadow directory, and it may not be possible to find the `my.dat` file relative to the source directory from there. Fat byte compilation files -------------------------- An earlier version of this PEP described "fat" Python byte code files. These files would contain the equivalent of multiple `pyc` files in a single `pyf` file, with a lookup table keyed off the appropriate magic number. This was an extensible file format so that the first 5 parallel Python implementations could be supported fairly efficiently, but with extension lookup tables available to scale `pyf` byte code objects as large as necessary. The fat byte compilation files were fairly complex, and inherently introduced difficult race conditions, so the current simplification of using directories was suggested. The same problem applies to using zip files as the fat pyc file format. Multiple file extensions ------------------------ The PEP author also considered an approach where multiple thin byte compiled files lived in the same place, but used different file extensions to designate the Python version. E.g. foo.pyc25, foo.pyc26, foo.pyc31 etc. This was rejected because of the clutter involved in writing so many different files. The multiple extension approach makes it more difficult (and an ongoing task) to update any tools that are dependent on the file extension. .pyc ---- A proposal was floated to call the `__pycache__` directory `.pyc` or some other dot-file name. This would have the effect on *nix systems of hiding the directory. There are many reasons why this was rejected by the BDFL [20]_ including the fact that dot-files are only special on some platforms, and we actually do *not* want to hide these completely from users. Reference implementation ======================== Work on this code is tracked in a Bazaar branch on Launchpad [22]_ until it's ready for merge into Python 3.2. The work-in-progress diff can also be viewed [23]_ and is updated automatically as new changes are uploaded. A Rietveld code review issue [24]_ has been opened as of 2010-04-01 (no, this is not an April Fools joke :). References ========== .. [1] PEP 3146 .. [2] The marshal module: http://www.python.org/doc/current/library/marshal.html .. [3] import.c: http://svn.python.org/view/python/branches/py3k/Python/import.c?view=markup .. [4] Ubuntu: .. [5] Debian: .. [6] Debian Python Policy: http://www.debian.org/doc/packaging-manuals/python-policy/ .. [7] PEP 384 .. [8] python-support: http://wiki.debian.org/DebianPythonFAQ#Whatispython-support.3F .. [9] python-central: http://wiki.debian.org/DebianPythonFAQ#Whatispython-central.3F .. [10] binascii.hexlify(): http://www.python.org/doc/current/library/binascii.html#binascii.hexlify .. [11] Jython: http://www.jython.org/ .. [12] IronPython: http://ironpython.net/ .. [13] PyPy: http://codespeak.net/pypy/dist/pypy/doc/ .. [14] Pynie: http://code.google.com/p/pynie/ .. [15] py_compile: http://docs.python.org/library/py_compile.html .. [16] compileall: http://docs.python.org/library/compileall.html .. [17] imp: http://www.python.org/doc/current/library/imp.html .. [18] PEP 302 .. [19] PEP 304 .. [20] http://www.mail-archive.com/python-dev at python.org/msg45203.html .. [21] importlib: http://docs.python.org/3.1/library/importlib.html .. [22] https://code.launchpad.net/~barry/python/pep3147 .. [23] https://code.launchpad.net/~barry/python/pep3147/+merge/22648 .. [24] http://codereview.appspot.com/842043/show ACKNOWLEDGMENTS =============== Barry Warsaw's original idea was for fat Python byte code files. Martin von Loewis reviewed an early draft of the PEP and suggested the simplification to store traditional `pyc` and `pyo` files in a directory. Many other people reviewed early versions of this PEP and provided useful feedback including but not limited to: * David Malcolm * Josselin Mouette * Matthias Klose * Michael Hudson * Michael Vogt * Piotr O?arowski * Scott Kitterman * Toshio Kuratomi Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From rnk at mit.edu Tue Apr 13 22:36:17 2010 From: rnk at mit.edu (Reid Kleckner) Date: Tue, 13 Apr 2010 16:36:17 -0400 Subject: [Python-Dev] Tuning Python dicts In-Reply-To: References: Message-ID: On Tue, Apr 13, 2010 at 12:12 PM, Daniel Stutzbach wrote: > I don't know what benchmarks were used to write dictnotes.txt, but moving > forward I would recommend implementing your changes on trunk (i.e., Python > 2.x) and running the Unladen Swallow Benchmarks, which you can get from the > link below: > > http://code.google.com/p/unladen-swallow/wiki/Benchmarks I'm a contributor, actually. ;) > They are macro-benchmarks on real applications.? You will probably also want > to write some micro-benchmarks of your own so that you can pinpoint any > bottlenecks in your code and determine where you are ahead of the current > dict implementation and where you are behind. What I really wanted to do was to find the benchmarks for the experiments discussed in dictnotex.txt so I could put them in the unladen benchmark repository, which now lives at hg.python.org. Since no one knows where they are, I think my next step will be to go back and see who wrote what parts of that file and contact them individually. Reid From martin at v.loewis.de Tue Apr 13 22:37:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Apr 2010 22:37:52 +0200 Subject: [Python-Dev] GSoC mentors needed Message-ID: <4BC4D620.7000401@v.loewis.de> It seems the Python (core) project needs some more mentors, to really match the proposals that we got. I don't want to post a list of proposals here, so if you are willing to potentially mentor a project, please join the soc2010-mentors list ASAP and/or contact Arc Riley. Doing so would neither be a commitment nor a guarantee that you will actually mentor a project - this depends on a lot of factors (your interest in a specific project, Google's ultimate assigment of slots, and so on). Regards, Martin From ncoghlan at gmail.com Tue Apr 13 23:04:32 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Apr 2010 07:04:32 +1000 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <20100413103835.1a4cfc09@heresy> References: <20100413021740.8C16FEE984@mail.python.org> <4BC46E03.6020601@gmail.com> <20100413100417.2bf4bbbc@heresy> <4BC47CE9.9030500@gmail.com> <20100413103835.1a4cfc09@heresy> Message-ID: <4BC4DC60.9030106@gmail.com> Barry Warsaw wrote: > On Apr 14, 2010, at 12:17 AM, Nick Coghlan wrote: > >> Barry Warsaw wrote: >>> On Apr 13, 2010, at 11:13 PM, Nick Coghlan wrote: >> Sounds reasonable. I ask because the various functions in runpy will >> also need to cover setting that value properly. > > Right. I'm looking at runpy now, but am less familiar with this code. Right > now I have anything executed with -m as setting __cached__ to None, which > seems right in the simple case because the -m doesn't appear to get > byte compiled. Is that correct? Yeah, the only time it uses byte-compiled files is if the original source is missing. Setting __cached__ to None for that case as well sounds like a reasonable starting point. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Apr 13 23:33:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Apr 2010 23:33:52 +0200 Subject: [Python-Dev] OS information, tags In-Reply-To: <20100413165827.07BD81FF350@kimball.webabinitio.net> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <87zl1730ut.fsf@uwakimon.sk.tsukuba.ac.jp> <4BC47DD3.60800@gmail.com> <20100413165827.07BD81FF350@kimball.webabinitio.net> Message-ID: <4BC4E340.60001@v.loewis.de> > That said, it's easier to add components and keywords than it is to > add a new selection box. I know how to do the former but not (yet) > the latter. It's not too difficult, either: edit schema.py to change the database (roundup will automatically sync with the database), then edit html/issue.item.html to display the field (pick one of the fields with a similar type, and copy-paste). Assuming this is an enumerated type, you actually need to start with creating a new Class (in schema.py). Ignore initial_data.py, and just define the class; then fill the enumerators through the web interface. This is one of the areas where roundup rocks (email follow-ups being the other one :-) One of the roundup downsides is access control. Take a similar field (access-control wise), and add the new field/class to all places where the similar field is used. That will make sure that any AC bugs in the original field will also spread to the new field :-( HTH, Martin From guido at python.org Wed Apr 14 01:44:49 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Apr 2010 16:44:49 -0700 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100413162107.171c724c@heresy> References: <20100413162107.171c724c@heresy> Message-ID: On Tue, Apr 13, 2010 at 1:21 PM, Barry Warsaw wrote: > I am attaching the latest revision of PEP 3147 to this message, which is also > available here: > > http://www.python.org/dev/peps/pep-3147/ > > I think the PEP is ready for pronouncement, and the patch is pretty much ready > for merging into py3k. ?The only thing that I can think of that is not > implemented yet is this section on PEP 302 loaders: > > ? ?PEP 302 [18]_ defined loaders have a `.get_filename()` method which > ? ?points to the `__file__` for a module. ?As part of this PEP, we will > ? ?extend this API, to include a new method `.get_paths()` which will > ? ?return a 2-tuple containing the path to the source file and the path > ? ?to where the matching `pyc` file is (or would be). > > I'm honestly not sure whether this is still essential, or whether the > importlib ABC changes Brett and I talked about at Pycon are still required. ?I > now believe they are at best a minor part of the implementation if so. ?Maybe > Brett can chime in on that. > > Everything else is implemented, tested, and has undergone four rounds of > Rietveld reviews (thanks Antoine, Benjamin, Brett, and Georg!). ?A fifth patch > set has been uploaded and is available here: > > http://codereview.appspot.com/842043/show > > This addresses all previous comments, includes some fixes from Brian Curtin > for Windows (thanks!) and fixes __main__ and -m support. ?I'd like to commit > this to py3k sooner rather than later so that we can shake out any additional > issues that might crop up, without having to continue to maintain my external > branches. > > Guido, what say you? Give me a couple of days; but I don't expect any problems given how the earlier discussion went. If you didn't hear from me by Friday go ahead and merge. > -Barry -- --Guido van Rossum (python.org/~guido) From victor.stinner at haypocalc.com Wed Apr 14 02:49:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 14 Apr 2010 02:49:00 +0200 Subject: [Python-Dev] Use surrogates for Python import Message-ID: <201004140249.00748.victor.stinner@haypocalc.com> Hi, Python3 refuses to start with LANG=C if it's installed in a non-ASCII directory. I wrote a patch fixing this issue, but it changes a lot of code and I would like your opinion. The main part changes import to use surrogateescape everywhere (find_module, load_source_module, null importer, zip importer, etc.). Other files have to be adapted: - traceback.c (tb_printinternal), - ast.c (ast_error_finish), - compile.c (compiler_error), - bltinmodule.c (builtin_compile), - _warnings.c (show_warning), - tokenizer.c (fp_setreadl), - ... It fixes also calculate_path(): sys.path may also contains directories using surrogates. Should I continue to work on this, or is it a bad idea to use surrogates in filenames used in Python imports? -- Related issue: #8242. The patch attached to this issue is a work-in-progress, I plan to cleanup it and to split it in small patches. It contains extra changes to fix many modules and tests if the current directory is not decodable. -- Victor Stinner http://www.haypocalc.com/ From chris.jerdonek at gmail.com Wed Apr 14 05:49:02 2010 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 13 Apr 2010 20:49:02 -0700 Subject: [Python-Dev] patch for review: unittest ImportError handling Message-ID: Hi folks, I have a patch to the unittest module for review here: http://bugs.python.org/issue7559#msg102801 (There have already been a couple rounds of discussion on how to best fix this.) This is my first patch, so any feedback is appreciated. Thanks, --Chris From steve at holdenweb.com Wed Apr 14 05:51:57 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 13 Apr 2010 23:51:57 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BA62546.10906@python.org> References: <4BA62546.10906@python.org> Message-ID: Why is it unavoidable that the Mac build will languish behind others? Are we supporting MacOs or aren't we? If we are, why isn't the creation of the build a part of the release process? Clearly it's not a priority given that nobody has seen fit to (or had time to) reply to this mail in three weeks. regards Steve webmaster at python.org wrote: > Hey all, > > This seems to happen whenever we do a new release (we've had a couple of > emails to webmaster at python.org about it since 2.6.5 was released). The > main download page for Python has a broken link for the Mac installer > (because it hasn't been built yet I assume): > > http://python.org/download/ > > The link 404s, with no explanation or alternate link - so for the casual > user who wants to install Python 2.6 on Mac OS X they are > sorely-out-of-luck. > > Not being able to provide a mac installer at the same time as other > platforms is one thing (and I accept that is unavoidable), breaking the > download links for Mac users for unspecified lengths of time is just bad > practise. If we create a new stable release without a Mac installer can > we at least provide a brief explanation and link to the *previous > version* until the new version is ready? > > All the best, > > Michael > > -------- Original Message -------- > Subject: Broken link to down > Date: Sun, 21 Mar 2010 13:40:36 +0000 > From: Ben Hodgson > To: webmaster at python.org > > > > Hey there, > > In case you don't know, the link on http://www.python.org/download/ to > the Python 2.6.5 Mac Installer Disk Image > (http://www.python.org/ftp/python/2.6.5/python-2.6.5_macosx10.3.dmg) is > broken. > > Cheers, > Ben > > -- > Ben Hodgson > http://benhodgson.com/ > -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From chris.jerdonek at gmail.com Wed Apr 14 05:54:39 2010 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 13 Apr 2010 20:54:39 -0700 Subject: [Python-Dev] patch for review: __import__ documentation Message-ID: Here is another patch for review: http://bugs.python.org/issue8370 This is a trivial fix to the 2.6 and 2.7 documentation. Thanks, --Chris From tseaver at palladion.com Wed Apr 14 05:57:01 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 13 Apr 2010 23:57:01 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> Message-ID: <4BC53D0D.2080802@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Holden wrote: > Why is it unavoidable that the Mac build will languish behind others? > Are we supporting MacOs or aren't we? If we are, why isn't the creation > of the build a part of the release process? > > Clearly it's not a priority given that nobody has seen fit to (or had > time to) reply to this mail in three weeks. Maybe the PSF should make it a priority by funding acquisition of the appropriate proprietary hardware (Mac / Windows) for the release manager. Otherwise the avaialbility of binaries is going to lag source releases forever. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvFPQ0ACgkQ+gerLs4ltQ5EkQCfTCMQTefF8dnjn2sZ8fHgJNNc HD8Ani13Nq8q1zaG8ttlpoZ/gZ3tQ3Ln =DuRD -----END PGP SIGNATURE----- From steve at holdenweb.com Wed Apr 14 06:07:29 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 00:07:29 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC53D0D.2080802@palladion.com> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> Message-ID: <4BC53F81.7050505@holdenweb.com> Tres Seaver wrote: > Steve Holden wrote: >> Why is it unavoidable that the Mac build will languish behind others? >> Are we supporting MacOs or aren't we? If we are, why isn't the creation >> of the build a part of the release process? > >> Clearly it's not a priority given that nobody has seen fit to (or had >> time to) reply to this mail in three weeks. > > Maybe the PSF should make it a priority by funding acquisition of the > appropriate proprietary hardware (Mac / Windows) for the release > manager. Otherwise the avaialbility of binaries is going to lag source > releases forever. > Thanks for your note. One of the reasons I took the trouble to comment on the issue was to determine whether resources are needed. Generally I regard it as the PSF's job to respond to requests for resources from the dev team, not to tell them how to do the work they already excel at. We did, a while ago, accept a hardware donation from Apple. If we need more Mac hardware I will be happy to ask the Infrastructure Committee to look into it, but I'd suggest we need guidance from someone actually involved in the build work to ensure we get maximum utility out of any investment. I do think it makes us look bad to have one supported platform lag the others, but it wasn't obvious to me whether hardware alone was the reason. If it is, the fix should be relatively simple. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From nad at acm.org Wed Apr 14 06:13:46 2010 From: nad at acm.org (Ned Deily) Date: Tue, 13 Apr 2010 21:13:46 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> Message-ID: In article , Steve Holden wrote: > Why is it unavoidable that the Mac build will languish behind others? > Are we supporting MacOs or aren't we? If we are, why isn't the creation > of the build a part of the release process? > > Clearly it's not a priority given that nobody has seen fit to (or had > time to) reply to this mail in three weeks. Wasn't that problem fixed weeks ago? The installer image has been available there since several days after the release. And the link seems fine now. -- Ned Deily, nad at acm.org From steve at holdenweb.com Wed Apr 14 06:17:23 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 00:17:23 -0400 Subject: [Python-Dev] Decimal <-> float comparisons in py3k. In-Reply-To: References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> <201003170932.09201.steve@pearwood.info> Message-ID: Guido van Rossum wrote: > On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano wrote: >> But mixed arithmetic runs into the problem, what do you want the result >> type to be? Given (say) decimal+float, returning either a Decimal or a >> float will be the wrong thing to do some of the time, so better to >> prohibit mixed arithmetic and let folks handle their own conversions. >> So +1 on continuing to prohibit mixed arithmetic. > > I'm not disagreeing, but I really wonder, what is the value of > supporting mixed comparisons then? Just because you *can* assign a > meaning to it doesn't mean you should. > > OTOH I'm sure a lot of people would like to see mixed arithmetic > supported, the PEP be damned, and they would probably be happy with > any simple rule about the return type even if it's not always ideal. I > note that there are cases where converting a long to a float also is > the wrong thing to do, and yet mixed long/float operations always > return floats. I suspect this latter behavior is a throwback to the days when conversion of an integer to a float was guaranteed not to cause overflow. With the extension of the integer type to longs that theory went out of the window, and with it the previously manageable "widening" that took place. > If you are amenable to this argument, I would propose > to make the result of mixed operations return a Decimal, since in some > "intuitive complexity" sense an int is a simpler type than a float and > a float is a simpler type than a Decimal -- so results return the more > complex type. But my intuition on this isn't super strong and I could > live with always returning a float as well -- there are always casts > to force the issue. > Alas Python now supports so many number systems that there is no longer a rational (in the non-numerical sense) ordering of the systems which allows a linear progression from one to the next. It therefore behooves us to consider the implications of such a decision *very* carefully. intuition alone (even yours, which I would back against most people's) may not suffice. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From martin at v.loewis.de Wed Apr 14 07:11:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 07:11:07 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> Message-ID: <4BC54E6B.3090209@v.loewis.de> > Why is it unavoidable that the Mac build will languish behind others? Because of lack of volunteers, and expertise (i.e. the experts lack time). > Are we supporting MacOs or aren't we? We aren't. Strictly speaking, "we" (python-dev) "support" nothing (in the sense that "we" can promise a support hotline, service contracts, or anything in that respect). That includes timely availability of stuff. If you want Python "support" on MacOS, contact Apple, ActiveState, or any other company that provides actual support. In a wider sense of "to support", MacOS is certainly supported by Python. There is everything in the source code that you need to make Python run on a Mac. Just download the sources and compile them yourself. > Clearly it's not a priority given that nobody has seen fit to (or had > time to) reply to this mail in three weeks. That is not surprising: none of the webmaster people would be able to answer the question. python-dev is indeed the right place to ask. Regards, Martin From martin at v.loewis.de Wed Apr 14 07:11:57 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 07:11:57 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC53D0D.2080802@palladion.com> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> Message-ID: <4BC54E9D.8080205@v.loewis.de> Tres Seaver wrote: > Steve Holden wrote: >> Why is it unavoidable that the Mac build will languish behind others? >> Are we supporting MacOs or aren't we? If we are, why isn't the creation >> of the build a part of the release process? > >> Clearly it's not a priority given that nobody has seen fit to (or had >> time to) reply to this mail in three weeks. > > Maybe the PSF should make it a priority by funding acquisition of the > appropriate proprietary hardware (Mac / Windows) for the release > manager. Otherwise the avaialbility of binaries is going to lag source > releases forever. Tres, can you be more explicit? How would such hardware help (whom specifically)? Regards, Martin From martin at v.loewis.de Wed Apr 14 07:14:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 07:14:55 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> Message-ID: <4BC54F4F.4090307@v.loewis.de> > Wasn't that problem fixed weeks ago? The installer image has been > available there since several days after the release. And the link > seems fine now. The inherent problem remains. There is no binary for 2.7b1, for example. The last binaries produced in the 2.7 testing process were for 2.7a2. Regards, Martin From steve at holdenweb.com Wed Apr 14 07:17:07 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 01:17:07 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC54E6B.3090209@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> Message-ID: <4BC54FD3.4030907@holdenweb.com> Martin v. L?wis wrote: >> Why is it unavoidable that the Mac build will languish behind others? > > Because of lack of volunteers, and expertise (i.e. the experts lack time). > >> Are we supporting MacOs or aren't we? > > We aren't. Strictly speaking, "we" (python-dev) "support" nothing (in > the sense that "we" can promise a support hotline, service contracts, or > anything in that respect). That includes timely availability of stuff. > If you want Python "support" on MacOS, contact Apple, ActiveState, or > any other company that provides actual support. > OK. I should confirm that I mean "support" in your wider sense below. > In a wider sense of "to support", MacOS is certainly supported by > Python. There is everything in the source code that you need to make > Python run on a Mac. Just download the sources and compile them yourself. > And yet we don't regard the Windows release as complete until you have built the binaries (for which service you deserve many thanks, by the way). Is the Mac platform one on which users will be happy to compile from source? I know its users are savvier than Windows users, and have a better tool set available to them, but they still seem to expect downloadable installers. >> Clearly it's not a priority given that nobody has seen fit to (or had >> time to) reply to this mail in three weeks. > > That is not surprising: none of the webmaster people would be able to > answer the question. python-dev is indeed the right place to ask. > I thought I'd picked this thread off python-dev. What point am I not understanding here? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From martin at v.loewis.de Wed Apr 14 07:36:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 07:36:25 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC54FD3.4030907@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> Message-ID: <4BC55459.6000904@v.loewis.de> >> In a wider sense of "to support", MacOS is certainly supported by >> Python. There is everything in the source code that you need to make >> Python run on a Mac. Just download the sources and compile them yourself. >> > And yet we don't regard the Windows release as complete until you have > built the binaries (for which service you deserve many thanks, by the way). This phenomenon exists for a lot of other systems, as well. For example, we also support Solaris, but stopped providing Solaris binaries since Python 1.5 (when I last built binaries for Das Python-Buch). People still can get Solaris binaries from ActiveState or Sunfreeware; Sun also ships Python as part of the system. > Is the Mac platform one on which users will be happy to compile from > source? I know its users are savvier than Windows users, and have a > better tool set available to them, but they still seem to expect > downloadable installers. The major difference in the "do it yourself" attitude is that Mac user get a compiler for free, as part of the operating system release, whereas for Windows, they have to pay for it (leaving alone VS Express for the moment). However, the real difference is motivation for contribution to open source projects. You normally contribute to scratch an itch. Unfortunately, these binaries don't come out such a motiviation. So the release manager roles are either altruistic, or rely on extrinsic motivations (money, reputation). >>> Clearly it's not a priority given that nobody has seen fit to (or had >>> time to) reply to this mail in three weeks. >> That is not surprising: none of the webmaster people would be able to >> answer the question. python-dev is indeed the right place to ask. >> > I thought I'd picked this thread off python-dev. What point am I not > understanding here? Maybe I misunderstood. I thought you copied it from the webmaster list (as the original To indicated). I don't recall having it seen on python-dev, but I kill many messages to python-dev without reading them. Regards, Martin From paul at rudin.co.uk Wed Apr 14 08:37:44 2010 From: paul at rudin.co.uk (Paul Rudin) Date: Wed, 14 Apr 2010 07:37:44 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> Message-ID: <87eiiibkuf.fsf@rudin.co.uk> "Martin v. L?wis" writes: > The major difference in the "do it yourself" attitude is that Mac user > get a compiler for free, as part of the operating system release, > whereas for Windows, they have to pay for it (leaving alone VS Express > for the moment). JOOI why ignore the express versions of the MS compilers? All (I think) MS compilers are available for free in command line versions - it's the GUI tools you pay for. From nad at acm.org Wed Apr 14 09:59:50 2010 From: nad at acm.org (Ned Deily) Date: Wed, 14 Apr 2010 00:59:50 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> Message-ID: In article <4BC54F4F.4090307 at v.loewis.de>, "Martin v. Lowis" wrote: > > Wasn't that problem fixed weeks ago? The installer image has been > > available there since several days after the release. And the link > > seems fine now. > > The inherent problem remains. There is no binary for 2.7b1, for example. > The last binaries produced in the 2.7 testing process were for 2.7a2. That's true. But there wouldn't be a traditional OS X installer for 2.7b1 anyway since it turns out it is not possible to build a multi-arch installer without patching because of a bug that wasn't caught before the code cutoff since there are no OS X buildbots that test that configuration. But, at the moment, there aren't any OS X buildbots at all, are there? That *is* something that the PSF could help with. I would be happy to help with that myself, although my time to do so will be very limited for the next few weeks. -- Ned Deily, nad at acm.org From fuzzyman at voidspace.org.uk Wed Apr 14 10:38:17 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 10:38:17 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC54E6B.3090209@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> Message-ID: <4BC57EF9.5040206@voidspace.org.uk> On 14/04/2010 07:11, "Martin v. L?wis" wrote: >> Why is it unavoidable that the Mac build will languish behind others? >> > Because of lack of volunteers, and expertise (i.e. the experts lack time). > > That doesn't explain why we leave a broken link in place when we do major releases - for days usually (if not weeks) with no explanation to users. That part of the release process is broken and should be fixed. Putting some placeholder text on the release page instead of the broken link is barely any more effort at all. For the last 2.6.5 release I changed the text and removed the dead link myself. When the Mac installer was uploaded someone else put the link back. [snip...] >> Clearly it's not a priority given that nobody has seen fit to (or had >> time to) reply to this mail in three weeks. >> > That is not surprising: none of the webmaster people would be able to > answer the question. python-dev is indeed the right place to ask. > Which is why I sent the email onto python-dev. However, no-one responded until Steve. All the best, Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 10:56:06 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 10:56:06 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> Message-ID: <4BC58326.4@voidspace.org.uk> On 14/04/2010 06:13, Ned Deily wrote: > In article, > Steve Holden wrote: > > >> Why is it unavoidable that the Mac build will languish behind others? >> Are we supporting MacOs or aren't we? If we are, why isn't the creation >> of the build a part of the release process? >> >> Clearly it's not a priority given that nobody has seen fit to (or had >> time to) reply to this mail in three weeks. >> > Wasn't that problem fixed weeks ago? The installer image has been > available there since several days after the release. And the link > seems fine now. > > The problem is the process that creates a new release with a 404 link to the Mac installer with no explanation. The 2.6.5 release (as always) caused several requests to webmaster from Mac users unable to download Python - which is a further waste of volunteer time as well as a cause of frustration for users. Building the Mac installer requires volunteer time which I'm not sure that more hardware will fix - compiling a full build of Python for Mac OS X (with all the Python modules like Tkinter etc) requires expertise which only a few people have. A Mac OS X machine (and location to keep it) for the buildbots is a *big* need however. All the best, Michael -- http://www.ironpythoninaction.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 11:07:42 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 11:07:42 +0200 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: References: Message-ID: <4BC585DE.9090804@voidspace.org.uk> On 14/04/2010 05:49, Chris Jerdonek wrote: > Hi folks, > > I have a patch to the unittest module for review here: > > http://bugs.python.org/issue7559#msg102801 > > (There have already been a couple rounds of discussion on how to best > fix this.) > > This is my first patch, so any feedback is appreciated. > I'm still not convinced that this isn't a backwards incompatible change - up until now, however horrible it may be, TestLoader.loadTestsFromName only raised an AttributeError when it failed to load a test. Changing it to allow it propagate ImportError means that code catching errors by handling AttributeError will be potentially broken by the fix. I'm certainly happy to discuss this here though. An alternative fix would be for a new API and deprecation of loadTestFromName. A new API could return a placeholder test that raises the original error when run - that way individual errors don't break the test collection phase but are still reported. This *could* be added to loadTestsFromName with an optional argument. By the way, in general please don't assign unittest bugs *away* from me on the tracker. I'm maintaining unittest and have been watching this issue. I haven't given it much attention recently because of getting the new features ready for the 2.7b1 release. I will certainly want to look through the patch before it is committed, if the consensus here is that changing this API to raise an ImportError is not a backwards incompatible change. All the best, Michael Foord > Thanks, > --Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 11:11:22 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 11:11:22 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC54FD3.4030907@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> Message-ID: <4BC586BA.9010704@voidspace.org.uk> On 14/04/2010 07:17, Steve Holden wrote: > [snip...] >> In a wider sense of "to support", MacOS is certainly supported by >> Python. There is everything in the source code that you need to make >> Python run on a Mac. Just download the sources and compile them yourself. >> >> > And yet we don't regard the Windows release as complete until you have > built the binaries (for which service you deserve many thanks, by the way). > > Is the Mac platform one on which users will be happy to compile from > source? I know its users are savvier than Windows users, and have a > better tool set available to them, but they still seem to expect > downloadable installers. > > Mac users definitely *do* expect installers. Building Python requires, I believe, the XCode development tools to be installed. Even then, building a full version of Python - with *all* the C extensions that are part of a Python release - is not a trivial task. All the best, Michael Foord >>> Clearly it's not a priority given that nobody has seen fit to (or had >>> time to) reply to this mail in three weeks. >>> >> That is not surprising: none of the webmaster people would be able to >> answer the question. python-dev is indeed the right place to ask. >> >> > I thought I'd picked this thread off python-dev. What point am I not > understanding here? > > regards > Steve > -- http://www.ironpythoninaction.com/ From robertc at robertcollins.net Wed Apr 14 11:16:04 2010 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Apr 2010 19:16:04 +1000 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: <4BC585DE.9090804@voidspace.org.uk> References: <4BC585DE.9090804@voidspace.org.uk> Message-ID: I'd really like to see a fix that works with loadTestsFromNames - generating failing tests, for instance, and the failing tests having the full import error string in them. This doesn't preclude raising ImportError from loadTestFromName, and in fact I'd encourage that as a step towards the aforementioned loadTestsFromNames change. It is a change with the existing misguided behaviour; sometimes thats better to do than preserving compat; as you say a new API might be best overall though. I'd go with a keyword arg rather than a new function, I think. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Apr 14 12:33:03 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 14 Apr 2010 06:33:03 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC54E9D.8080205@v.loewis.de> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC54E9D.8080205@v.loewis.de> Message-ID: <4BC599DF.8060207@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: > Tres Seaver wrote: >> Steve Holden wrote: >>> Why is it unavoidable that the Mac build will languish behind others? >>> Are we supporting MacOs or aren't we? If we are, why isn't the creation >>> of the build a part of the release process? >>> Clearly it's not a priority given that nobody has seen fit to (or had >>> time to) reply to this mail in three weeks. >> Maybe the PSF should make it a priority by funding acquisition of the >> appropriate proprietary hardware (Mac / Windows) for the release >> manager. Otherwise the avaialbility of binaries is going to lag source >> releases forever. > > Tres, > > can you be more explicit? How would such hardware help (whom specifically)? I assumed that creation of installer binaries for a release depends on having the release manager or a lieutenant have access to the given platform (Windows, OS/X) and tools, For instance, the RM or lieutenant might only have access to such a platform part-time (e.g., only while at work, or only at home). In such a case, providing additional hardware could expedite creation of the binaries. I'm sorry if I misunderstood the issue. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvFmdkACgkQ+gerLs4ltQ5B/QCglQlANbOgn8BodKowku/aCBc0 QdgAoLQi2Ntlns/B4FdxNs0kCHC5VVqk =Xm+Q -----END PGP SIGNATURE----- From chris.jerdonek at gmail.com Wed Apr 14 12:54:29 2010 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 14 Apr 2010 03:54:29 -0700 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: <4BC585DE.9090804@voidspace.org.uk> References: <4BC585DE.9090804@voidspace.org.uk> Message-ID: On Wed, Apr 14, 2010 at 2:07 AM, Michael Foord wrote: > I'm still not convinced that this isn't a backwards incompatible change - up > until now, however horrible it may be, TestLoader.loadTestsFromName only > raised an AttributeError when it failed to load a test. Changing it to allow > it propagate ImportError means that code catching errors by handling > AttributeError will be potentially broken by the fix. I'm certainly happy to > discuss this here though. Thanks for your response. Michael. To understand your position better, would you view changing the AttributeError in *any* way an incompatible change (e.g. changing just the error message), or is it only changing the error type that you view as backwards incompatible? The reason I ask is that I think it's the change in what the error contains (e.g. the error message and stack trace) that is the more important part of this change -- rather than whether that information be wrapped in an ImportError versus an AttributeError. It is the information rather than the particular error type that will assist an end-developer in diagnosing future unit-test failures. > An alternative fix would be for a new API and deprecation of > loadTestFromName. A new API could return a placeholder test that raises the > original error when run - that way individual errors don't break the test > collection phase but are still reported. This *could* be added to > loadTestsFromName with an optional argument. I'm not opposed to adding a new API, but I think it would be valuable if we could find a way for people to enjoy the benefit of not having the stack trace swallowed -- without having to change their code. I'm currently part of a project where the current behavior makes it harder to track down unit test failures. I imagine many developers are in a similar situation. This seems to be a bug, and such developers may be in a position where they can't change how their unit tests are run, but they're nevertheless stuck diagnosing the failures. I'm writing on the assumption that there is a way to preserve the stack trace and error message of an ImportError while re-raising it as an AttributeError. > By the way, in general please don't assign unittest bugs *away* from me on > the tracker. Will do. I hadn't come across any guidance re: assigning in the development workflow documentation. Maybe we should add a cautionary note about this in the documentation somewhere. In general, do all reports stay assigned to the maintainer (if a maintainer exists), or is it a per-maintainer preference on how that is handled? Thanks, --Chris From p.f.moore at gmail.com Wed Apr 14 12:59:09 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Apr 2010 11:59:09 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <87eiiibkuf.fsf@rudin.co.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> Message-ID: On 14 April 2010 07:37, Paul Rudin wrote: > "Martin v. L?wis" writes: > >> The major difference in the "do it yourself" attitude is that Mac user >> get a compiler for free, as part of the operating system release, >> whereas for Windows, they have to pay for it (leaving alone VS Express >> for the moment). > > JOOI why ignore the express versions of the MS compilers? All (I think) > MS compilers are available for free in command line versions - it's the > GUI tools you pay for. I believe that the express editions don't include some of the advanced optimisations (profile guided optimisation rings a bell) which are used to build the official binaries. So if the binaries were built using Express Edition, they would be somewhat slower. That is just my recollection, however - it may be out of date or wrong. Paul. From fuzzyman at voidspace.org.uk Wed Apr 14 13:12:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 13:12:21 +0200 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: References: <4BC585DE.9090804@voidspace.org.uk> Message-ID: <4BC5A315.6000005@voidspace.org.uk> On 14/04/2010 12:54, Chris Jerdonek wrote: > On Wed, Apr 14, 2010 at 2:07 AM, Michael Foord > wrote: > >> I'm still not convinced that this isn't a backwards incompatible change - up >> until now, however horrible it may be, TestLoader.loadTestsFromName only >> raised an AttributeError when it failed to load a test. Changing it to allow >> it propagate ImportError means that code catching errors by handling >> AttributeError will be potentially broken by the fix. I'm certainly happy to >> discuss this here though. >> > Thanks for your response. Michael. To understand your position > better, would you view changing the AttributeError in *any* way an > incompatible change (e.g. changing just the error message), or is it > only changing the error type that you view as backwards incompatible? > > The reason I ask is that I think it's the change in what the error > contains (e.g. the error message and stack trace) that is the more > important part of this change -- rather than whether that information > be wrapped in an ImportError versus an AttributeError. It is the > information rather than the particular error type that will assist an > end-developer in diagnosing future unit-test failures. > Changing the error message to provide more useful information, possibly including the original traceback, would certainly avoid the potential for incompatibility. I'd be interested in seeing what other folks here on python-dev think. > >> An alternative fix would be for a new API and deprecation of >> loadTestFromName. A new API could return a placeholder test that raises the >> original error when run - that way individual errors don't break the test >> collection phase but are still reported. This *could* be added to >> loadTestsFromName with an optional argument. >> > I'm not opposed to adding a new API, but I think it would be valuable > if we could find a way for people to enjoy the benefit of not having > the stack trace swallowed -- without having to change their code. It's a double edged sword though - it means existing users depending on the fact that this API only raises AttributeError have to change *their* code. For a new API I like Rob Collin's suggestion of a keyword argument to loadTestsFromNames that returns an error placeholder instead of raising an exception. That couldn't be put into 2.7 now though. > I'm > currently part of a project where the current behavior makes it harder > to track down unit test failures. I imagine many developers are in a > similar situation. This seems to be a bug, and such developers may be > in a position where they can't change how their unit tests are run, > but they're nevertheless stuck diagnosing the failures. > > I'm writing on the assumption that there is a way to preserve the > stack trace and error message of an ImportError while re-raising it as > an AttributeError. > > The original stacktrace could be included as part of the error message. Pretty horrible though. >> By the way, in general please don't assign unittest bugs *away* from me on >> the tracker. >> > Will do. I hadn't come across any guidance re: assigning in the > development workflow documentation. Maybe we should add a cautionary > note about this in the documentation somewhere. In general, do all > reports stay assigned to the maintainer (if a maintainer exists), or > is it a per-maintainer preference on how that is handled? > > Where an issue is assigned to an existing maintainer I would wait for a response on the issue tracker or raise it here on python-dev rather than re-assigning the issue. In general issues corresponding to modules that have an active maintainer should be assigned to the maintainer. It is particularly an issue for me with unittest because I want to track changes in unittest in the unittest2 package and need to know about all changes. I'm often around on #python-dev to discuss things. All the best, Michael Foord > Thanks, > --Chris > -- http://www.ironpythoninaction.com/ From chris.jerdonek at gmail.com Wed Apr 14 14:31:44 2010 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 14 Apr 2010 05:31:44 -0700 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: <4BC5A315.6000005@voidspace.org.uk> References: <4BC585DE.9090804@voidspace.org.uk> <4BC5A315.6000005@voidspace.org.uk> Message-ID: On Wed, Apr 14, 2010 at 4:12 AM, Michael Foord wrote: > On 14/04/2010 12:54, Chris Jerdonek wrote: >> >> On Wed, Apr 14, 2010 at 2:07 AM, Michael Foord >> ?wrote: >> >>> >>> I'm still not convinced that this isn't a backwards incompatible change - >>> up >>> until now, however horrible it may be, TestLoader.loadTestsFromName only >>> raised an AttributeError when it failed to load a test. Changing it to >>> allow >>> it propagate ImportError means that code catching errors by handling >>> AttributeError will be potentially broken by the fix. I'm certainly happy >>> to >>> discuss this here though. >>> >> >> Thanks for your response. Michael. ?To understand your position >> better, would you view changing the AttributeError in *any* way an >> incompatible change (e.g. changing just the error message), or is it >> only changing the error type that you view as backwards incompatible? >> >> The reason I ask is that I think it's the change in what the error >> contains (e.g. the error message and stack trace) that is the more >> important part of this change -- rather than whether that information >> be wrapped in an ImportError versus an AttributeError. ?It is the >> information rather than the particular error type that will assist an >> end-developer in diagnosing future unit-test failures. >> > > Changing the error message to provide more useful information, possibly > including the original traceback, would certainly avoid the potential for > incompatibility. I'd be interested in seeing what other folks here on > python-dev think. > >> >>> >>> An alternative fix would be for a new API and deprecation of >>> loadTestFromName. A new API could return a placeholder test that raises >>> the >>> original error when run - that way individual errors don't break the test >>> collection phase but are still reported. This *could* be added to >>> loadTestsFromName with an optional argument. >>> >> >> I'm not opposed to adding a new API, but I think it would be valuable >> if we could find a way for people to enjoy the benefit of not having >> the stack trace swallowed -- without having to change their code. > > It's a double edged sword though - it means existing users depending on the > fact that this API only raises AttributeError have to change *their* code. > > For a new API I like Rob Collin's suggestion of a keyword argument to > loadTestsFromNames that returns an error placeholder instead of raising an > exception. That couldn't be put into 2.7 now though. > >> I'm >> currently part of a project where the current behavior makes it harder >> to track down unit test failures. ?I imagine many developers are in a >> similar situation. ?This seems to be a bug, and such developers may be >> in a position where they can't change how their unit tests are run, >> but they're nevertheless stuck diagnosing the failures. >> >> I'm writing on the assumption that there is a way to preserve the >> stack trace and error message of an ImportError while re-raising it as >> an AttributeError. >> >> > > The original stacktrace could be included as part of the error message. > Pretty horrible though. I just experimented with this myself. Couldn't we do something like the following to change the error type and preserve the stack trace -- without including the stack trace as part of the message? try: foo() except ImportError: excType, excValue, excTraceback = sys.exc_info() raise AttributeError, excValue, excTraceback Superficially this seems to have the desired effect. We could also adjust the error text to indicate that the AttributeError was originally an ImportError. The above is similar to the code you referenced in one of your comments to issue 7559-- http://twistedmatrix.com/trac/browser/trunk/twisted/python/reflect.py?rev=28196#L384 --Chris From barry at python.org Wed Apr 14 13:46:24 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 07:46:24 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC58326.4@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> Message-ID: <20100414074624.1ce1d2cf@heresy> On Apr 14, 2010, at 10:56 AM, Michael Foord wrote: >The problem is the process that creates a new release with a 404 link to >the Mac installer with no explanation. The 2.6.5 release (as always) >caused several requests to webmaster from Mac users unable to download >Python - which is a further waste of volunteer time as well as a cause >of frustration for users. As the RM, that's my fault then. When I start creating the page for a new release I will leave the Windows and Mac links commented out. I do (now ) wait for Martin to upload the Windows installers before announcing the release, but generally don't wait for Ronald to produce the Mac images, so I leave those commented out too. When Ronald builds the Mac image and provides it to me, I'll upload them and tweak the page. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Apr 14 13:49:34 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 07:49:34 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <20100414074934.21666b15@heresy> On Apr 13, 2010, at 04:44 PM, Guido van Rossum wrote: >Give me a couple of days; but I don't expect any problems given how >the earlier discussion went. If you didn't hear from me by Friday go >ahead and merge. Thanks Guido. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Apr 14 13:51:20 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 07:51:20 -0400 Subject: [Python-Dev] [Python-checkins] r80025 - peps/trunk/pep-3147.txt In-Reply-To: <4BC4DC60.9030106@gmail.com> References: <20100413021740.8C16FEE984@mail.python.org> <4BC46E03.6020601@gmail.com> <20100413100417.2bf4bbbc@heresy> <4BC47CE9.9030500@gmail.com> <20100413103835.1a4cfc09@heresy> <4BC4DC60.9030106@gmail.com> Message-ID: <20100414075120.0446d10e@heresy> On Apr 14, 2010, at 07:04 AM, Nick Coghlan wrote: >Yeah, the only time it uses byte-compiled files is if the original >source is missing. Setting __cached__ to None for that case as well >sounds like a reasonable starting point. Cool, thanks. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Wed Apr 14 14:45:43 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 14:45:43 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414074624.1ce1d2cf@heresy> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <20100414074624.1ce1d2cf@heresy> Message-ID: <4BC5B8F7.2080504@voidspace.org.uk> On 14/04/2010 13:46, Barry Warsaw wrote: > On Apr 14, 2010, at 10:56 AM, Michael Foord wrote: > > >> The problem is the process that creates a new release with a 404 link to >> the Mac installer with no explanation. The 2.6.5 release (as always) >> caused several requests to webmaster from Mac users unable to download >> Python - which is a further waste of volunteer time as well as a cause >> of frustration for users. >> > As the RM, that's my fault then. When I start creating the page for a new > release I will leave the Windows and Mac links commented out. I do (now > ) wait for Martin to upload the Windows installers before announcing the > release, but generally don't wait for Ronald to produce the Mac images, so I > leave those commented out too. When Ronald builds the Mac image and provides > it to me, I'll upload them and tweak the page. > Can we amend that to having some placeholder text saying that the Mac installer is not yet available and a link to the previous available version please. That can then be replaced with the normal link once the Mac installer is uploaded. Thanks Michael Foord > -Barry > -- http://www.ironpythoninaction.com/ From barry at python.org Wed Apr 14 13:58:25 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 07:58:25 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5B8F7.2080504@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <20100414074624.1ce1d2cf@heresy> <4BC5B8F7.2080504@voidspace.org.uk> Message-ID: <20100414075825.7d39d940@heresy> On Apr 14, 2010, at 02:45 PM, Michael Foord wrote: >Can we amend that to having some placeholder text saying that the Mac >installer is not yet available and a link to the previous available >version please. That can then be replaced with the normal link once the >Mac installer is uploaded. You mean, on the x.y.z release page, or a separate page? I guess you're saying that it's better to include some "not available yet" text on the x.y.x release page than to just comment out the platform. That makes sense, and I'd be happy to do that, since I already make such a caveat in the release announcement. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ctb at msu.edu Wed Apr 14 15:39:36 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 14 Apr 2010 06:39:36 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC599DF.8060207@palladion.com> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC54E9D.8080205@v.loewis.de> <4BC599DF.8060207@palladion.com> Message-ID: <20100414133936.GB25826@idyll.org> On Wed, Apr 14, 2010 at 06:33:03AM -0400, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Martin v. L?wis wrote: > > Tres Seaver wrote: > >> Steve Holden wrote: > >>> Why is it unavoidable that the Mac build will languish behind others? > >>> Are we supporting MacOs or aren't we? If we are, why isn't the creation > >>> of the build a part of the release process? > >>> Clearly it's not a priority given that nobody has seen fit to (or had > >>> time to) reply to this mail in three weeks. > >> Maybe the PSF should make it a priority by funding acquisition of the > >> appropriate proprietary hardware (Mac / Windows) for the release > >> manager. Otherwise the avaialbility of binaries is going to lag source > >> releases forever. > > > > Tres, > > > > can you be more explicit? How would such hardware help (whom specifically)? > > I assumed that creation of installer binaries for a release depends on > having the release manager or a lieutenant have access to the given > platform (Windows, OS/X) and tools, For instance, the RM or lieutenant > might only have access to such a platform part-time (e.g., only while at > work, or only at home). In such a case, providing additional hardware > could expedite creation of the binaries. As with Windows, I personally find that building Python with all the associated libraries is blocked on getting the right libraries installed, not on getting the compilers (which are available to us for free) ... but I'm sure that easy access to hardware is an issue, too. I can provide command-line access to a Mac OS X machine but I'm not sure that's enough. Let me know if anyone wants that. Separately, I'd be happy to put forward a proposal to the PSF to fund RMs and their lieutenants with a Mac or a PC, whichever they needed to keep things moving. It's the least "we" can do, IMO, and hardware is just not that expensive compared to the dedication of the volunteers. If Georg, Benjamin, Martin, or Ronald are interested, please just tell me (or Steve, or the PSF board, or ...) what you want and I'll work on getting it funded. --titus -- C. Titus Brown, ctb at msu.edu From fuzzyman at voidspace.org.uk Wed Apr 14 15:47:23 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 15:47:23 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414075825.7d39d940@heresy> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <20100414074624.1ce1d2cf@heresy> <4BC5B8F7.2080504@voidspace.org.uk> <20100414075825.7d39d940@heresy> Message-ID: <4BC5C76B.2010805@voidspace.org.uk> On 14/04/2010 13:58, Barry Warsaw wrote: > On Apr 14, 2010, at 02:45 PM, Michael Foord wrote: > > >> Can we amend that to having some placeholder text saying that the Mac >> installer is not yet available and a link to the previous available >> version please. That can then be replaced with the normal link once the >> Mac installer is uploaded. >> > You mean, on the x.y.z release page, or a separate page? I guess you're > saying that it's better to include some "not available yet" text on the x.y.x > release page than to just comment out the platform. That makes sense, and I'd > be happy to do that, since I already make such a caveat in the release > announcement. > > Yes, I mean on the release page. The issue is that the download links on the sidebar / front page go straight to the latest release page. If there isn't yet a Mac installer available, and no alternative link to get the previous version, it leaves Mac users with no obvious way to get a Mac installer *at all*. (Those who know the site well can find the previous versions themselves, but for users not intimately familiar with our site layout it is 'a challenge'.) Thanks Michael > -Barry > -- http://www.ironpythoninaction.com/ From ctb at msu.edu Wed Apr 14 15:50:06 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 14 Apr 2010 06:50:06 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC55459.6000904@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> Message-ID: <20100414135006.GE25826@idyll.org> On Wed, Apr 14, 2010 at 07:36:25AM +0200, "Martin v. L?wis" wrote: > >> In a wider sense of "to support", MacOS is certainly supported by > >> Python. There is everything in the source code that you need to make > >> Python run on a Mac. Just download the sources and compile them yourself. > >> > > And yet we don't regard the Windows release as complete until you have > > built the binaries (for which service you deserve many thanks, by the way). > > This phenomenon exists for a lot of other systems, as well. For example, > we also support Solaris, but stopped providing Solaris binaries since > Python 1.5 (when I last built binaries for Das Python-Buch). People > still can get Solaris binaries from ActiveState or Sunfreeware; Sun also > ships Python as part of the system. I personally think the Mac is pretty important, as one of the big three consumer operating systems... > > Is the Mac platform one on which users will be happy to compile from > > source? I know its users are savvier than Windows users, and have a > > better tool set available to them, but they still seem to expect > > downloadable installers. > > The major difference in the "do it yourself" attitude is that Mac user > get a compiler for free, as part of the operating system release, > whereas for Windows, they have to pay for it (leaving alone VS Express > for the moment). Actually, I think the more pernicious factor is that a version of Python comes pre-installed on Mac OS X, which means the up-front demand is lower for a pre-compiled version. This is problematic, though, because that version of Python only gets upgraded with full releases of Mac OS X (which are not very well correlated with releases of Python, of course). So we have lots of Python installs out there that, in the absence of a precompiled binary version, can't be upgraded without installing the developer tools. > However, the real difference is motivation for contribution to open > source projects. You normally contribute to scratch an itch. > Unfortunately, these binaries don't come out such a motiviation. So the > release manager roles are either altruistic, or rely on extrinsic > motivations (money, reputation). I don't know what to do about motivation but if there are barriers that we can lower, please let me know. cheers, --titus -- C. Titus Brown, ctb at msu.edu From ijmorlan at uwaterloo.ca Wed Apr 14 15:52:01 2010 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Wed, 14 Apr 2010 09:52:01 -0400 (EDT) Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100413162107.171c724c@heresy> References: <20100413162107.171c724c@heresy> Message-ID: On Tue, 13 Apr 2010, Barry Warsaw wrote: > I am attaching the latest revision of PEP 3147 to this message, which is > also available here: > > http://www.python.org/dev/peps/pep-3147/ [....] > PEP: 3147 > Title: PYC Repository Directories [....] > Further, pyc files will contain a magic string that differentiates the > Python version they were compiled for. This allows multiple byte > compiled cache files to co-exist for a single Python source file. > > This scheme has the added benefit of reducing the clutter in a Python > package directory. > > When a Python source file is imported for the first time, a > `__pycache__` directory will be created in the package directory, if > one does not already exist. The pyc file for the imported source will > be written to the `__pycache__` directory, using the magic-tag > formatted name. If either the creation of the `__pycache__` directory > or the pyc file inside that fails, the import will still succeed, just > as it does in a pre-PEP-3147 world. [....] Thank you for doing the work on this improvement. I have one wording suggestion which I hope isn't bikeshedding: up above, I think the sentence containing "pyc files will contain a magic string" would be clearer if it made it clear that the file *names*, not (just?) the file contents, will contain the magic tag. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From barry at python.org Wed Apr 14 16:00:14 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 10:00:14 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414133936.GB25826@idyll.org> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC54E9D.8080205@v.loewis.de> <4BC599DF.8060207@palladion.com> <20100414133936.GB25826@idyll.org> Message-ID: <20100414100014.1846e099@heresy> On Apr 14, 2010, at 06:39 AM, C. Titus Brown wrote: >Separately, I'd be happy to put forward a proposal to the PSF to fund RMs and >their lieutenants with a Mac or a PC, whichever they needed to keep things >moving. It's the least "we" can do, IMO, and hardware is just not that >expensive compared to the dedication of the volunteers. If Georg, Benjamin, >Martin, or Ronald are interested, please just tell me (or Steve, or the PSF >board, or ...) what you want and I'll work on getting it funded. While I appreciate the offer, I don't think this is as useful as it could be. Speaking as an RM, I would happily build all three releases (source tarballs, Windows installers and Mac disk images) if I had easy access to the necessary machines and environments over the 'net, and precise instructions on how to do the builds. Ideally, those instructions would be "run this script" or even rolled into the current release.py script that is used to build the tarballs. I have Macs and a Windows machine. The Windows machine is not easily accessible as it's a dual-boot with Ubuntu. As stated previously, the difficult part is getting all the necessary dependencies in place and the expertise to know the steps that are required to build. A network accessible machine for Mac and Windows, where other experts such as Martin and Roland can help maintain and configure, would be ideal. That way, and of the platform experts or RM could push a button and build the installers. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Apr 14 16:33:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 00:33:15 +1000 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <4BC5D22B.3070408@gmail.com> Isaac Morland wrote: > I have one wording suggestion which I hope isn't bikeshedding: up above, > I think the sentence containing "pyc files will contain a magic string" > would be clearer if it made it clear that the file *names*, not (just?) > the file contents, will contain the magic tag. That's not bikeshedding, that's picking up a mistake in the PEP :) Indeed, the magic tag only goes in the file names (the pyc files themselves contain the corresponding magic number, just as they always have). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Apr 14 16:40:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 00:40:39 +1000 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5C76B.2010805@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <20100414074624.1ce1d2cf@heresy> <4BC5B8F7.2080504@voidspace.org.uk> <20100414075825.7d39d940@heresy> <4BC5C76B.2010805@voidspace.org.uk> Message-ID: <4BC5D3E7.3050806@gmail.com> Michael Foord wrote: > Yes, I mean on the release page. The issue is that the download links on > the sidebar / front page go straight to the latest release page. If > there isn't yet a Mac installer available, and no alternative link to > get the previous version, it leaves Mac users with no obvious way to get > a Mac installer *at all*. (Those who know the site well can find the > previous versions themselves, but for users not intimately familiar with > our site layout it is 'a challenge'.) I would actually use the "source only" release pages as a guide here. They provide a link to the download page for the last version that provided Mac and Windows binaries. This makes it clear to the downloader that the binaries are for an older version, while still making a binary version easy to find. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Apr 14 16:47:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 00:47:50 +1000 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC54F4F.4090307@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> Message-ID: <4BC5D596.9030303@gmail.com> Martin v. L?wis wrote: >> Wasn't that problem fixed weeks ago? The installer image has been >> available there since several days after the release. And the link >> seems fine now. > > The inherent problem remains. There is no binary for 2.7b1, for example. > The last binaries produced in the 2.7 testing process were for 2.7a2. I seem to recall there was some issue (aside from the current lack of a reliable OS X buildbot) that prevented us from regularly grabbing the head of the tree and using it to automatically build the Windows and Mac installers (to check that the installers could actually be created, preventing a repeat of the Mac situation with 2.7b1). Am I misremembering the discussion from last time this topic came up? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Apr 14 16:53:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 00:53:08 +1000 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: <4BC5A315.6000005@voidspace.org.uk> References: <4BC585DE.9090804@voidspace.org.uk> <4BC5A315.6000005@voidspace.org.uk> Message-ID: <4BC5D6D4.2050907@gmail.com> Michael Foord wrote: > Changing the error message to provide more useful information, possibly > including the original traceback, would certainly avoid the potential > for incompatibility. I'd be interested in seeing what other folks here > on python-dev think. Without looking at the details, my question is whether this can be ignored in 2.x and fixed for 3.x by setting __cause__ appropriately (i.e. by using the "raise from" syntax). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ctb at msu.edu Wed Apr 14 16:57:03 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 14 Apr 2010 07:57:03 -0700 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC5D596.9030303@gmail.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC5D596.9030303@gmail.com> Message-ID: <20100414145703.GF8041@idyll.org> On Thu, Apr 15, 2010 at 12:47:50AM +1000, Nick Coghlan wrote: > Martin v. L?wis wrote: > >> Wasn't that problem fixed weeks ago? The installer image has been > >> available there since several days after the release. And the link > >> seems fine now. > > > > The inherent problem remains. There is no binary for 2.7b1, for example. > > The last binaries produced in the 2.7 testing process were for 2.7a2. > > I seem to recall there was some issue (aside from the current lack of a > reliable OS X buildbot) that prevented us from regularly grabbing the > head of the tree and using it to automatically build the Windows and Mac > installers (to check that the installers could actually be created, > preventing a repeat of the Mac situation with 2.7b1). > > Am I misremembering the discussion from last time this topic came up? Making the Windows binary build process automatic and robust is challenging if you hate Windows (like I do). Martin also made the point that it's been broken forever, so people don't seem to care :). ISTR Martin just makes them manually. It's on my TODO list to robustify this process. OTOH, my Mac automated builds/tests are working just fine. I could produce nightly binaries from those easily enough: http://lyorn.idyll.org/ctb/pb-dev/p/python/ Just need to figure out the magic doohickey commands... will add to list. cheers, --titus -- C. Titus Brown, ctb at msu.edu From janssen at parc.com Wed Apr 14 17:23:55 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 08:23:55 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> Message-ID: <41208.1271258635@parc.com> Ned Deily wrote: > In article <4BC54F4F.4090307 at v.loewis.de>, > "Martin v. Lowis" wrote: > > > > Wasn't that problem fixed weeks ago? The installer image has been > > > available there since several days after the release. And the link > > > seems fine now. > > > > The inherent problem remains. There is no binary for 2.7b1, for example. > > The last binaries produced in the 2.7 testing process were for 2.7a2. > > That's true. But there wouldn't be a traditional OS X installer for > 2.7b1 anyway since it turns out it is not possible to build a multi-arch > installer without patching because of a bug that wasn't caught before > the code cutoff since there are no OS X buildbots that test that > configuration. But, at the moment, there aren't any OS X buildbots at > all, are there? That *is* something that the PSF could help with. I > would be happy to help with that myself I'd be happy to help where I can, too. All my automated testing of UpLib (Windows, Ubuntu, Fedora, OS X) is done on Apple servers running OS X and VirtualBox and Hudson, so I've got some experience there. Bill From janssen at parc.com Wed Apr 14 17:36:35 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 08:36:35 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC58326.4@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> Message-ID: <41372.1271259395@parc.com> Michael Foord wrote: > Building the Mac installer requires volunteer time which I'm not sure > that more hardware will fix - compiling a full build of Python for Mac > OS X (with all the Python modules like Tkinter etc) requires expertise > which only a few people have. That's nuts. Why isn't this expertise captured in the form of a script? > A Mac OS X machine (and location to keep it) for the buildbots is a > *big* need however. At least two. You want Leopard and Snow Leopard, too. Bill From fuzzyman at voidspace.org.uk Wed Apr 14 17:41:13 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 17:41:13 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <41372.1271259395@parc.com> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> Message-ID: <4BC5E219.8030509@voidspace.org.uk> On 14/04/2010 17:36, Bill Janssen wrote: > Michael Foord wrote: > > >> Building the Mac installer requires volunteer time which I'm not sure >> that more hardware will fix - compiling a full build of Python for Mac >> OS X (with all the Python modules like Tkinter etc) requires expertise >> which only a few people have. >> > That's nuts. Why isn't this expertise captured in the form of a script? > > >> A Mac OS X machine (and location to keep it) for the buildbots is a >> *big* need however. >> > At least two. You want Leopard and Snow Leopard, too. > Well - an XServe that we can run virtualisation on would be the *ideal* solution. I think the X serves are the only machines you are *allowed* to virtualise OS X on (?). All the best, Michael > Bill > -- http://www.ironpythoninaction.com/ From skip at pobox.com Wed Apr 14 17:51:47 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 10:51:47 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> Message-ID: <19397.58515.517331.130360@montanaro.dyndns.org> Steve> Why is it unavoidable that the Mac build will languish behind Steve> others? Are we supporting MacOs or aren't we? If we are, why Steve> isn't the creation of the build a part of the release process? Steve> Clearly it's not a priority given that nobody has seen fit to (or Steve> had time to) reply to this mail in three weeks. I'm not sure who normally creates the Mac distribution, perhaps Ronald Ousorren? It would appear that either Ronald (or whoever) has been unavailable or there was no coordination between the Mac and non-Mac folks involved in the release. I'm willing to give it a whirl (I have both a MacBook Pro and a PowerMac G5 at home - both running Max OSX 10.5.x (Leopard)) though I will almost certainly need a cheat sheet for the process. I normally treat my Macs as Unix boxes from a Python perspective so don't make framework builds. Skip From skip at pobox.com Wed Apr 14 17:54:46 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 10:54:46 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC53F81.7050505@holdenweb.com> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC53F81.7050505@holdenweb.com> Message-ID: <19397.58694.802388.707092@montanaro.dyndns.org> Steve> I do think it makes us look bad to have one supported platform Steve> lag the others, but it wasn't obvious to me whether hardware Steve> alone was the reason. If it is, the fix should be relatively Steve> simple. I can't believe it's a hardware issue. Probably half the people with laptops at the last PyCon I attended were Macs. I suspect anyone with a recent Mac running Leopard or Snow Leopard should be able to build the binary release. It's probably just a matter of knowing how. Skip From fuzzyman at voidspace.org.uk Wed Apr 14 17:58:28 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 17:58:28 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5E219.8030509@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> <4BC5E219.8030509@voidspace.org.uk> Message-ID: <4BC5E624.9020207@voidspace.org.uk> On 14/04/2010 17:41, Michael Foord wrote: > [snip...] >>> A Mac OS X machine (and location to keep it) for the buildbots is a >>> *big* need however. >> At least two. You want Leopard and Snow Leopard, too. > > Well - an XServe that we can run virtualisation on would be the > *ideal* solution. I think the X serves are the only machines you are > *allowed* to virtualise OS X on (?). An alternative that solves the problem of finding somewhere to put the machine would be to use a service like Mac Mini colocation hosting: http://www.macminicolo.net/ That could be used both for debugging OS X specific problems *and* as a buildbot. All the best, Michael > > All the best, > > Michael > >> Bill > > -- http://www.ironpythoninaction.com/ From skip at pobox.com Wed Apr 14 17:59:16 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 10:59:16 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> Message-ID: <19397.58964.821434.965051@montanaro.dyndns.org> Ned> ... since there are no OS X buildbots that test that configuration. Ned> But, at the moment, there aren't any OS X buildbots at all, are Ned> there? That *is* something that the PSF could help with.... What happened to the big-ass computer farm for Python which was being put together by someone at (I think) Michigan State? Skip From barry at python.org Wed Apr 14 18:04:42 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 12:04:42 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <19397.58515.517331.130360@montanaro.dyndns.org> References: <4BA62546.10906@python.org> <19397.58515.517331.130360@montanaro.dyndns.org> Message-ID: <20100414120442.5bba807e@heresy> On Apr 14, 2010, at 10:51 AM, skip at pobox.com wrote: > Steve> Why is it unavoidable that the Mac build will languish behind > Steve> others? Are we supporting MacOs or aren't we? If we are, why > Steve> isn't the creation of the build a part of the release process? > > Steve> Clearly it's not a priority given that nobody has seen fit to (or > Steve> had time to) reply to this mail in three weeks. > >I'm not sure who normally creates the Mac distribution, perhaps Ronald >Ousorren? It would appear that either Ronald (or whoever) has been >unavailable or there was no coordination between the Mac and non-Mac folks >involved in the release. > >I'm willing to give it a whirl (I have both a MacBook Pro and a PowerMac G5 >at home - both running Max OSX 10.5.x (Leopard)) though I will almost >certainly need a cheat sheet for the process. I normally treat my Macs as >Unix boxes from a Python perspective so don't make framework builds. From the RM perspective, what I would really like to see is updates to the release.py script to check dependencies and automate as much as possible, as well as updates to PEP 101 for any process steps that can't be automated. This goes for both Windows and OS X. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From skip at pobox.com Wed Apr 14 18:01:34 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 11:01:34 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC586BA.9010704@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> Message-ID: <19397.59102.929541.18939@montanaro.dyndns.org> Michael> Mac users definitely *do* expect installers. Building Python Michael> requires, I believe, the XCode development tools to be Michael> installed. XCode is free, and I suspect many people have it (I do). Michael> Even then, building a full version of Python - with *all* the C Michael> extensions that are part of a Python release - is not a trivial Michael> task. Isn't that just a matter of having the recipe written down somewhere? Skip From fuzzyman at voidspace.org.uk Wed Apr 14 18:18:01 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 18:18:01 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <19397.59102.929541.18939@montanaro.dyndns.org> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> Message-ID: <4BC5EAB9.3030306@voidspace.org.uk> On 14/04/2010 18:01, skip at pobox.com wrote: > Michael> Mac users definitely *do* expect installers. Building Python > Michael> requires, I believe, the XCode development tools to be > Michael> installed. > > XCode is free, and I suspect many people have it (I do). > Sure - but probably not your average Python-on-Mac user. Or at least a good proportion of them, particularly newbies who we are keen to keep the experience of obtaining Python simple. First download and then install 1gigabyte of developer tools (seriously) requiring registration, then compile Python from source yourself, is not the way to go. (And yes the XCode tools are included in the OS disks - but not the latest versions, especially if you are still running Leopard.) > Michael> Even then, building a full version of Python - with *all* the C > Michael> extensions that are part of a Python release - is not a trivial > Michael> task. > > Isn't that just a matter of having the recipe written down somewhere? > Yes, that would be nice. :-) Preferably a recipe that doesn't involve Macports or Fink which some of us are allergic to. Michael > Skip > -- http://www.ironpythoninaction.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 18:23:03 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 18:23:03 +0200 Subject: [Python-Dev] patch for review: unittest ImportError handling In-Reply-To: <4BC5D6D4.2050907@gmail.com> References: <4BC585DE.9090804@voidspace.org.uk> <4BC5A315.6000005@voidspace.org.uk> <4BC5D6D4.2050907@gmail.com> Message-ID: <4BC5EBE7.1@voidspace.org.uk> On 14/04/2010 16:53, Nick Coghlan wrote: > Michael Foord wrote: > >> Changing the error message to provide more useful information, possibly >> including the original traceback, would certainly avoid the potential >> for incompatibility. I'd be interested in seeing what other folks here >> on python-dev think. >> > Without looking at the details, my question is whether this can be > ignored in 2.x and fixed for 3.x by setting __cause__ appropriately > (i.e. by using the "raise from" syntax). > Yes, definitely - if maintaining the exception as an AttributeError is determined to be the correct course of action for that API. Chris showed an alternative way of achieving the same effect for Python 2, but that is possibly moot given that 2.7b1 is out (unless this is an acceptable bugfix to include in b2). All the best, Michael > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ From p.f.moore at gmail.com Wed Apr 14 18:34:07 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Apr 2010 17:34:07 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414120442.5bba807e@heresy> References: <4BA62546.10906@python.org> <19397.58515.517331.130360@montanaro.dyndns.org> <20100414120442.5bba807e@heresy> Message-ID: On 14 April 2010 17:04, Barry Warsaw wrote: > From the RM perspective, what I would really like to see is updates to > the release.py script to check dependencies and automate as much as possible, > as well as updates to PEP 101 for any process steps that can't be automated. > > This goes for both Windows and OS X. >From what I recall, the PC build process is pretty much routine (I can't recall how much it's scripted, and how much it's manual, but well-documented and simple, steps). I don't know what extra is needed to build the final installer, but I'd be willing to have a go at testing the existing process and if necessary adding some automation. Paul. From janssen at parc.com Wed Apr 14 18:37:34 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 09:37:34 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5EAB9.3030306@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> Message-ID: <42790.1271263054@parc.com> Michael Foord wrote: > > Isn't that just a matter of having the recipe written down somewhere? > > > > Yes, that would be nice. :-) Preferably a recipe that doesn't involve > Macports or Fink which some of us are allergic to. Yes, ditto the MacPorts/Fink allergy. All we need is a script, right? The released branches should be built automatically every night anyway, just for regression testing. Perhaps we should take this up on Mac-Python? Bill From trent at snakebite.org Wed Apr 14 18:34:34 2010 From: trent at snakebite.org (Trent Nelson) Date: Wed, 14 Apr 2010 17:34:34 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <19397.58964.821434.965051@montanaro.dyndns.org> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <19397.58964.821434.965051@montanaro.dyndns.org> Message-ID: <001001cadbf0$6253bf10$26fb3d30$@snakebite.org> > What happened to the big-ass computer farm for Python which was > being put together by someone at (I think) Michigan State? That sounds a lot like Snakebite (www.snakebite.org), which is still... uhhh, a work in progress ;-) We've run into an issue recently that's thwarted progress, but that'll hopefully be resolved in the next couple of weeks. And then... full steam ahead! Trent. From steve at holdenweb.com Wed Apr 14 18:49:46 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 12:49:46 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <42790.1271263054@parc.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> <42790.1271263054@parc.com> Message-ID: <4BC5F22A.9010807@holdenweb.com> Bill Janssen wrote: > Michael Foord wrote: > >>> Isn't that just a matter of having the recipe written down somewhere? >>> >> Yes, that would be nice. :-) Preferably a recipe that doesn't involve >> Macports or Fink which some of us are allergic to. > > Yes, ditto the MacPorts/Fink allergy. > > All we need is a script, right? The released branches should be built > automatically every night anyway, just for regression testing. > > Perhaps we should take this up on Mac-Python? > Please do, and let me know if resources of some kind would help. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 18:52:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 18:52:46 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5F22A.9010807@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> <42790.1271263054@parc.com> <4BC5F22A.9010807@holdenweb.com> Message-ID: <4BC5F2DE.9070307@voidspace.org.uk> On 14/04/2010 18:49, Steve Holden wrote: > Bill Janssen wrote: > >> Michael Foord wrote: >> >> >>>> Isn't that just a matter of having the recipe written down somewhere? >>>> >>>> >>> Yes, that would be nice. :-) Preferably a recipe that doesn't involve >>> Macports or Fink which some of us are allergic to. >>> >> Yes, ditto the MacPorts/Fink allergy. >> >> All we need is a script, right? The released branches should be built >> automatically every night anyway, just for regression testing. >> >> Perhaps we should take this up on Mac-Python? >> >> > Please do, and let me know if resources of some kind would help. > A Mac buildbot would *definitely* be useful and I know some of the Python-dev crew would like access to a Mac OS X machine for trying out fixes when none of the core Mac maintainers are around. A couple of co-located, or otherwise hosted, Mac boxes would be very useful. The obvious question is who would hold the keys (maintain the buildbot)? I don't know if Ronald has the spare capacity to do this (?). If someone will help me with the initial setup I can otherwise volunteer. All the best, Michael > regards > Steve > -- http://www.ironpythoninaction.com/ From ctb at msu.edu Wed Apr 14 18:52:49 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 14 Apr 2010 09:52:49 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <42790.1271263054@parc.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> <42790.1271263054@parc.com> Message-ID: <20100414165249.GB3952@idyll.org> On Wed, Apr 14, 2010 at 09:37:34AM -0700, Bill Janssen wrote: > Michael Foord wrote: > > > > Isn't that just a matter of having the recipe written down somewhere? > > > > > > > Yes, that would be nice. :-) Preferably a recipe that doesn't involve > > Macports or Fink which some of us are allergic to. > > Yes, ditto the MacPorts/Fink allergy. > > All we need is a script, right? The released branches should be built > automatically every night anyway, just for regression testing. Please pass it along to me, when you get it working... I build python2.7 nightly on Mac OS X, but just at the command line. see: http://lyorn.idyll.org/ctb/pb-dev/p/python/ http://lyorn.idyll.org/ctb/pb-dev/p/python/show_all (the Windows build is flaky for me, so the 'show_all' shows mostly Windows builds). --titus -- C. Titus Brown, ctb at msu.edu From ronaldoussoren at mac.com Wed Apr 14 18:01:19 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Apr 2010 18:01:19 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <155072217312148274731633653972868664038-Webmail@me.com> References: <155072217312148274731633653972868664038-Webmail@me.com> Message-ID: <42735872743376525492939197280196737792-Webmail@me.com> On Wednesday, April 14, 2010, at 04:47PM, "Nick Coghlan" wrote: >Martin v. L?wis wrote: >>> Wasn't that problem fixed weeks ago? The installer image has been >>> available there since several days after the release. And the link >>> seems fine now. >> >> The inherent problem remains. There is no binary for 2.7b1, for example. >> The last binaries produced in the 2.7 testing process were for 2.7a2. That's because I haven't had time yet to actually build the installer for OSX due to lack of free time because both my professional and private lives are fully filled at the moment (which is great, but leaves very little time to hack on Python or PyObjC). > >I seem to recall there was some issue > (aside from the current lack of a >reliable OS X buildbot) Speaking of which... I have a mac-mini that could be used for a buildbot. How much work is needed to kickstart a buildbot, taking into account that I'd prefer to have a buildbot with different configure-flags that the default unix build (that is, I want to test a framework build that is a universal binary). that prevented us from regularly grabbing the >head of the tree and using it to automatically build the Windows and Mac >installers (to check that the installers could actually be created, >preventing a repeat of the Mac situation with 2.7b1). Creating the Mac installer is easy: just run Mac/BuildScript/build-installer.py on an OSX 10.5 system where a local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The system should also not have fink or darwinports and a clean /usr/local tree to avoid contaminating the build. Ronald From ctb at msu.edu Wed Apr 14 19:05:54 2010 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 14 Apr 2010 10:05:54 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5F2DE.9070307@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> <42790.1271263054@parc.com> <4BC5F22A.9010807@holdenweb.com> <4BC5F2DE.9070307@voidspace.org.uk> Message-ID: <20100414170550.GA24153@idyll.org> On Wed, Apr 14, 2010 at 06:52:46PM +0200, Michael Foord wrote: > On 14/04/2010 18:49, Steve Holden wrote: >> Bill Janssen wrote: >> >>> Michael Foord wrote: >>> >>> >>>>> Isn't that just a matter of having the recipe written down somewhere? >>>>> >>>>> >>>> Yes, that would be nice. :-) Preferably a recipe that doesn't involve >>>> Macports or Fink which some of us are allergic to. >>>> >>> Yes, ditto the MacPorts/Fink allergy. >>> >>> All we need is a script, right? The released branches should be built >>> automatically every night anyway, just for regression testing. >>> >>> Perhaps we should take this up on Mac-Python? >>> >>> >> Please do, and let me know if resources of some kind would help. >> > > A Mac buildbot would *definitely* be useful and I know some of the > Python-dev crew would like access to a Mac OS X machine for trying out > fixes when none of the core Mac maintainers are around. A couple of > co-located, or otherwise hosted, Mac boxes would be very useful. > > The obvious question is who would hold the keys (maintain the buildbot)? > I don't know if Ronald has the spare capacity to do this (?). If someone > will help me with the initial setup I can otherwise volunteer. We have some space in our machine room, and some sysadmins that like open source. I will ask them if they are willing to do physical maintenance (profs wisely aren't allowed access to the machine room). That would really be ideal... I will report back to interested people. --titus -- C. Titus Brown, ctb at msu.edu From trent at snakebite.org Wed Apr 14 19:18:49 2010 From: trent at snakebite.org (Trent Nelson) Date: Wed, 14 Apr 2010 18:18:49 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <001001cadbf0$6253bf10$26fb3d30$@snakebite.org> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <19397.58964.821434.965051@montanaro.dyndns.org> <001001cadbf0$6253bf10$26fb3d30$@snakebite.org> Message-ID: <001701cadbf6$911a7b30$b34f7190$@snakebite.org> > > What happened to the big-ass computer farm for Python which was > > being put together by someone at (I think) Michigan State? > > That sounds a lot like Snakebite (www.snakebite.org), which is > still... uhhh, a work in progress ;-) Actually, for those that are interested, here's a copy of the presentation I gave at the Testing in Python session at PyCon a few months ago: http://www.snakebite.org/presentations/snakebite-pycon2010-tip.pptx (Office 2007-2010) http://www.snakebite.org/presentations/snakebite-pycon2010-tip.ppt (Office 97-2003) If anything, it'll shed some light on all the unforeseen issues we've been running into since the project's inception. The presentation is a little out of date -- I spent three months earlier this year on the network and it's definitely in the most respectable state it's been in yet. Coupla' photos for those that are interested: http://snakebite.org/images/IMG_4384.JPG http://snakebite.org/images/IMG_4392.JPG http://snakebite.org/images/IMG_4393.JPG http://snakebite.org/images/IMG_4394.JPG http://snakebite.org/images/IMG_4395.JPG http://snakebite.org/images/IMG_4396.JPG http://snakebite.org/images/IMG_4401.JPG http://snakebite.org/images/IMG_4402.JPG http://snakebite.org/images/IMG_4403.JPG http://snakebite.org/images/IMG_4405.JPG http://snakebite.org/images/IMG_4410.JPG http://snakebite.org/images/IMG_4418.JPG http://snakebite.org/images/IMG_4424.JPG http://snakebite.org/images/IMG_4425.JPG We've got three racks filled to the brim with all sorts of servers: - 4xItanium 2 @ 1.5GHz, 16GB RAM, HP-UX 11iv3 - 4xItanium 2 @ 1.5GHz, 30GB RAM, RHEL 5.3 - 2xUltraSPARC III 900MHz, 8GB, Solaris 10 (file/zfs/nfs server -- 16x146GB 2Gb FC) - 2xUltraSPARC III 1.2GHz, 4GB, Solaris 10 - 2xPA-RISC 875MHz, 8GB, HP-UX 11iv1 - 4 AIX boxes w/ 2x1.5GHz, 8GB, AIX 5.1, 5.2, 5.3 & 6.1 - 10 dedicated VMware x86/64 boxes, ranging from dual core 8GB to 8 core monsters with 64GB - 4x667MHz AlphaServer, 8GB, Tru64 - 4x600MHz SGI Octane 300, IRIX 6.22 - ....and lots of other stuff. Actually, the only platform we don't have is Mac OS X. Although I've got a contact at Apple that I'll start harassing again once I'm back in East Lansing. Trent. From janssen at parc.com Wed Apr 14 19:23:48 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 10:23:48 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5F2DE.9070307@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> <42790.1271263054@parc.com> <4BC5F22A.9010807@holdenweb.com> <4BC5F2DE.9070307@voidspace.org.uk> Message-ID: <44886.1271265828@parc.com> Michael Foord wrote: > On 14/04/2010 18:49, Steve Holden wrote: > > Bill Janssen wrote: > > > >> Michael Foord wrote: > >> > >> > >>>> Isn't that just a matter of having the recipe written down somewhere? > >>>> > >>>> > >>> Yes, that would be nice. :-) Preferably a recipe that doesn't involve > >>> Macports or Fink which some of us are allergic to. > >>> > >> Yes, ditto the MacPorts/Fink allergy. > >> > >> All we need is a script, right? The released branches should be built > >> automatically every night anyway, just for regression testing. > >> > >> Perhaps we should take this up on Mac-Python? > >> > >> > > Please do, and let me know if resources of some kind would help. > > > > A Mac buildbot would *definitely* be useful and I know some of the > Python-dev crew would like access to a Mac OS X machine for trying out > fixes when none of the core Mac maintainers are around. A couple of > co-located, or otherwise hosted, Mac boxes would be very useful. Yes! > > The obvious question is who would hold the keys (maintain the > buildbot)? I don't know if Ronald has the spare capacity to do this > (?). If someone will help me with the initial setup I can otherwise > volunteer. Sure, I can help with that. Bill From steve at holdenweb.com Wed Apr 14 19:25:20 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 13:25:20 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC58326.4@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> Message-ID: <4BC5FA80.5000808@holdenweb.com> Michael Foord wrote: > On 14/04/2010 06:13, Ned Deily wrote: >> In article, >> Steve Holden wrote: >> >> >>> Why is it unavoidable that the Mac build will languish behind others? >>> Are we supporting MacOs or aren't we? If we are, why isn't the creation >>> of the build a part of the release process? >>> >>> Clearly it's not a priority given that nobody has seen fit to (or had >>> time to) reply to this mail in three weeks. >>> >> Wasn't that problem fixed weeks ago? The installer image has been >> available there since several days after the release. And the link >> seems fine now. >> >> > The problem is the process that creates a new release with a 404 link to > the Mac installer with no explanation. The 2.6.5 release (as always) > caused several requests to webmaster from Mac users unable to download > Python - which is a further waste of volunteer time as well as a cause > of frustration for users. > > Building the Mac installer requires volunteer time which I'm not sure > that more hardware will fix - compiling a full build of Python for Mac > OS X (with all the Python modules like Tkinter etc) requires expertise > which only a few people have. > > A Mac OS X machine (and location to keep it) for the buildbots is a > *big* need however. > How about as a first step the release build process include a check for broken links before committing the web content for a new release? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Wed Apr 14 19:25:20 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 13:25:20 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC58326.4@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> Message-ID: <4BC5FA80.5000808@holdenweb.com> Michael Foord wrote: > On 14/04/2010 06:13, Ned Deily wrote: >> In article, >> Steve Holden wrote: >> >> >>> Why is it unavoidable that the Mac build will languish behind others? >>> Are we supporting MacOs or aren't we? If we are, why isn't the creation >>> of the build a part of the release process? >>> >>> Clearly it's not a priority given that nobody has seen fit to (or had >>> time to) reply to this mail in three weeks. >>> >> Wasn't that problem fixed weeks ago? The installer image has been >> available there since several days after the release. And the link >> seems fine now. >> >> > The problem is the process that creates a new release with a 404 link to > the Mac installer with no explanation. The 2.6.5 release (as always) > caused several requests to webmaster from Mac users unable to download > Python - which is a further waste of volunteer time as well as a cause > of frustration for users. > > Building the Mac installer requires volunteer time which I'm not sure > that more hardware will fix - compiling a full build of Python for Mac > OS X (with all the Python modules like Tkinter etc) requires expertise > which only a few people have. > > A Mac OS X machine (and location to keep it) for the buildbots is a > *big* need however. > How about as a first step the release build process include a check for broken links before committing the web content for a new release? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Wed Apr 14 19:27:19 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 19:27:19 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FA80.5000808@holdenweb.com> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <4BC5FA80.5000808@holdenweb.com> Message-ID: <4BC5FAF7.7040109@voidspace.org.uk> On 14/04/2010 19:25, Steve Holden wrote: > Michael Foord wrote: > >> On 14/04/2010 06:13, Ned Deily wrote: >> >>> In article, >>> Steve Holden wrote: >>> >>> >>> >>>> Why is it unavoidable that the Mac build will languish behind others? >>>> Are we supporting MacOs or aren't we? If we are, why isn't the creation >>>> of the build a part of the release process? >>>> >>>> Clearly it's not a priority given that nobody has seen fit to (or had >>>> time to) reply to this mail in three weeks. >>>> >>>> >>> Wasn't that problem fixed weeks ago? The installer image has been >>> available there since several days after the release. And the link >>> seems fine now. >>> >>> >>> >> The problem is the process that creates a new release with a 404 link to >> the Mac installer with no explanation. The 2.6.5 release (as always) >> caused several requests to webmaster from Mac users unable to download >> Python - which is a further waste of volunteer time as well as a cause >> of frustration for users. >> >> Building the Mac installer requires volunteer time which I'm not sure >> that more hardware will fix - compiling a full build of Python for Mac >> OS X (with all the Python modules like Tkinter etc) requires expertise >> which only a few people have. >> >> A Mac OS X machine (and location to keep it) for the buildbots is a >> *big* need however. >> >> > How about as a first step the release build process include a check for > broken links before committing the web content for a new release? > > Yes - needed but orthogonal. :-) Michael > regards > Steve > -- http://www.ironpythoninaction.com/ From steve at holdenweb.com Wed Apr 14 19:36:04 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 13:36:04 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> Message-ID: <4BC5FD04.8060208@holdenweb.com> Paul Moore wrote: > On 14 April 2010 07:37, Paul Rudin wrote: >> "Martin v. L?wis" writes: >> >>> The major difference in the "do it yourself" attitude is that Mac user >>> get a compiler for free, as part of the operating system release, >>> whereas for Windows, they have to pay for it (leaving alone VS Express >>> for the moment). >> JOOI why ignore the express versions of the MS compilers? All (I think) >> MS compilers are available for free in command line versions - it's the >> GUI tools you pay for. > > I believe that the express editions don't include some of the advanced > optimisations (profile guided optimisation rings a bell) which are > used to build the official binaries. So if the binaries were built > using Express Edition, they would be somewhat slower. > > That is just my recollection, however - it may be out of date or wrong. > Paul. I spent some considerable effort last year ensuring the developer community was well-supplied with MS developer licenses that give access to any necessary tools. Was I wasting my time? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Wed Apr 14 19:36:04 2010 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Apr 2010 13:36:04 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> Message-ID: <4BC5FD04.8060208@holdenweb.com> Paul Moore wrote: > On 14 April 2010 07:37, Paul Rudin wrote: >> "Martin v. L?wis" writes: >> >>> The major difference in the "do it yourself" attitude is that Mac user >>> get a compiler for free, as part of the operating system release, >>> whereas for Windows, they have to pay for it (leaving alone VS Express >>> for the moment). >> JOOI why ignore the express versions of the MS compilers? All (I think) >> MS compilers are available for free in command line versions - it's the >> GUI tools you pay for. > > I believe that the express editions don't include some of the advanced > optimisations (profile guided optimisation rings a bell) which are > used to build the official binaries. So if the binaries were built > using Express Edition, they would be somewhat slower. > > That is just my recollection, however - it may be out of date or wrong. > Paul. I spent some considerable effort last year ensuring the developer community was well-supplied with MS developer licenses that give access to any necessary tools. Was I wasting my time? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From skip at pobox.com Wed Apr 14 19:41:18 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 12:41:18 -0500 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <42735872743376525492939197280196737792-Webmail@me.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> Message-ID: <19397.65086.457847.833679@montanaro.dyndns.org> Ronald> Creating the Mac installer is easy: just run Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a Ronald> local version of Tcl/Tk 8.4 is installed in Ronald> /Library/Frameworks. The system should also not have fink or Ronald> darwinports and a clean /usr/local tree to avoid contaminating Ronald> the build. For those of us who live in an X11 world and use tools which Apple doesn't provide, like it or not, Fink or MacPorts are often a practical necessity. Would it be sufficient to modify the environment so that /sw, /opt/local and /usr/local don't appear in any paths when the script is run? On my laptop (MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are currently "polluted" with /opt/local. MANPATH and PATH contain /usr/local. Skip From barry at python.org Wed Apr 14 20:01:03 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 14:01:03 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FD04.8060208@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> <4BC5FD04.8060208@holdenweb.com> Message-ID: <20100414140103.1915f471@heresy> On Apr 14, 2010, at 01:36 PM, Steve Holden wrote: >I spent some considerable effort last year ensuring the developer >community was well-supplied with MS developer licenses that give access >to any necessary tools. Was I wasting my time? At the time I didn't care because I had no access to anything Windows. Now I do, though it's a bit of a PITA (because I have to reboot my main dev box). How can I get a license and the tools? Installing the express version was pretty easy, though it has limitations of course. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From janssen at parc.com Wed Apr 14 20:01:23 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 11:01:23 PDT Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19397.65086.457847.833679@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> Message-ID: <45888.1271268083@parc.com> skip at pobox.com wrote: > > Ronald> Creating the Mac installer is easy: just run > Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a > Ronald> local version of Tcl/Tk 8.4 is installed in > Ronald> /Library/Frameworks. The system should also not have fink or > Ronald> darwinports and a clean /usr/local tree to avoid contaminating > Ronald> the build. > > For those of us who live in an X11 world and use tools which Apple doesn't > provide, like it or not, Actually, Skip, you're talking about my world there (X11 and GNU Emacs and Gimp and ...), and I don't use Fink or MacPorts on my Macs. > Fink or MacPorts are often a practical necessity. Fink is deadly, MacPorts much more benign, in my experience. Which is several years out-of-date, before I realized I didn't need either one of them, and before the UNIX community started adding configure patches to support OS X builds more widely. Perhaps they've improved. In any case, they shouldn't be needed on buildbots maintained by the PSF. > Would it be sufficient to modify the environment so that /sw, /opt/local and > /usr/local don't appear in any paths when the script is run? On my laptop > (MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are > currently "polluted" with /opt/local. MANPATH and PATH contain /usr/local. Probably fine on your personal Mac. And the build scripts can probably mask those out on their own. Bill From brian.curtin at gmail.com Wed Apr 14 20:03:17 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 14 Apr 2010 13:03:17 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FD04.8060208@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> <4BC5FD04.8060208@holdenweb.com> Message-ID: On Wed, Apr 14, 2010 at 12:36, Steve Holden wrote: > Paul Moore wrote: > > On 14 April 2010 07:37, Paul Rudin wrote: > >> "Martin v. L?wis" writes: > >> > >>> The major difference in the "do it yourself" attitude is that Mac user > >>> get a compiler for free, as part of the operating system release, > >>> whereas for Windows, they have to pay for it (leaving alone VS Express > >>> for the moment). > >> JOOI why ignore the express versions of the MS compilers? All (I think) > >> MS compilers are available for free in command line versions - it's the > >> GUI tools you pay for. > > > > I believe that the express editions don't include some of the advanced > > optimisations (profile guided optimisation rings a bell) which are > > used to build the official binaries. So if the binaries were built > > using Express Edition, they would be somewhat slower. > > > > That is just my recollection, however - it may be out of date or wrong. > > Paul. > > I spent some considerable effort last year ensuring the developer > community was well-supplied with MS developer licenses that give access > to any necessary tools. Was I wasting my time? > > regards > Steve > I certainly hope not. I believe full MSDN licenses were involved there, which are a great resource to a Windows developer. I'll have a whack at building the Windows installer. It would be nice to see it integrated with the build process, including testing of an installed version. We already have one or more Linux buildbots which test an installed build, and that setup uncovered an additional bug in a recent fix of mine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Apr 14 20:06:44 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 14 Apr 2010 14:06:44 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FD04.8060208@holdenweb.com> References: =?US-ASCII?Q?=3C4BA62546=2E10906=40python=2Eorg=3E=0D=0A=09=3Chq3?= =?US-ASCII?Q?e52=248oj=241=40dough=2Egmane=2Eorg=3E=09=3C4BC54E6B?= =?US-ASCII?Q?=2E3090209=40v=2Eloewis=2Ede=3E=0D=0A=09=3C4BC54FD3=2E4?= =?US-ASCII?Q?030907=40holdenweb=2Ecom=3E=09=3C4BC55459=2E60009?= =?US-ASCII?Q?04=40v=2Eloewis=2Ede=3E=0D=0A=09=3C87eiiibkuf=2Efsf=40?= =?US-ASCII?Q?rudin=2Eco=2Euk=3E=0D=0A=09=3Cm2p79990c6b1004140359?= =?US-ASCII?Q?ze789ae7n6d7a2bffba5bd9be=40mail=2Egmail?= =?US-ASCII?Q?=2Ecom=3E_=3C4BC5FD04=2E8060208=40holdenweb=2Ecom=3E?= Message-ID: <742c849c-6fe9-435a-9cbe-aaba084fd654@email.android.com> In my case it was not a waste of time. I use MSDN for dev and testing. Just not for release building. "Steve Holden" wrote: >Paul Moore wrote: >> On 14 April 2010 07:37, Paul Rudin wrote: >>> "Martin v. L?wis" writes: >>> >>>> The major difference in the "do it yourself" attitude is that Mac user >>>> get a compiler for free, as part of the operating system release, >>>> whereas for Windows, they have to pay for it (leaving alone VS Express >>>> for the moment). >>> JOOI why ignore the express versions of the MS compilers? All (I think) >>> MS compilers are available for free in command line versions - it's the >>> GUI tools you pay for. >> >> I believe that the express editions don't include some of the advanced >> optimisations (profile guided optimisation rings a bell) which are >> used to build the official binaries. So if the binaries were built >> using Express Edition, they would be somewhat slower. >> >> That is just my recollection, however - it may be out of date or wrong. >> Paul. > >I spent some considerable effort last year ensuring the developer >community was well-supplied with MS developer licenses that give access >to any necessary tools. Was I wasting my time? > >regards > Steve >-- >Steve Holden +1 571 484 6266 +1 800 494 3119 >See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ >Holden Web LLC http://www.holdenweb.com/ >UPCOMING EVENTS: http://holdenweb.eventbrite.com/ >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com > -- Eric. From barry at python.org Wed Apr 14 20:09:54 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 14:09:54 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <20100414140954.5385d281@heresy> On Apr 14, 2010, at 09:52 AM, Isaac Morland wrote: >I have one wording suggestion which I hope isn't bikeshedding: up above, I >think the sentence containing "pyc files will contain a magic string" >would be clearer if it made it clear that the file *names*, not (just?) >the file contents, will contain the magic tag. Yep, that clarifies things. Thanks. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Wed Apr 14 20:21:31 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:21:31 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC586BA.9010704@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> Message-ID: <4BC607AB.4080203@v.loewis.de> > Mac users definitely *do* expect installers. Building Python requires, I > believe, the XCode development tools to be installed. Even then, > building a full version of Python - with *all* the C extensions that are > part of a Python release - is not a trivial task. The same is true for any other operating system, though: you need to install the compiler tool chain (sometimes, you need to buy it first), and compiling Python with all extensions is not a trivial task. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:24:36 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:24:36 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC599DF.8060207@palladion.com> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC54E9D.8080205@v.loewis.de> <4BC599DF.8060207@palladion.com> Message-ID: <4BC60864.7030107@v.loewis.de> > I assumed that creation of installer binaries for a release depends on > having the release manager or a lieutenant have access to the given > platform (Windows, OS/X) and tools, For instance, the RM or lieutenant > might only have access to such a platform part-time (e.g., only while at > work, or only at home). In such a case, providing additional hardware > could expedite creation of the binaries. > > I'm sorry if I misunderstood the issue. I think you do misunderstand: the people that do (occasionally, or eventually) provide OSX binaries have hardware available to them all the time. They only have part-time to work on Python, though. Of course, if somebody would promise a more timely provision of binaries, and can't do that because of lack of hardware, we would have to look for a solution. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:35:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:35:02 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414135006.GE25826@idyll.org> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <20100414135006.GE25826@idyll.org> Message-ID: <4BC60AD6.40101@v.loewis.de> > I personally think the Mac is pretty important, as one of the big three > consumer operating systems... [...] > I don't know what to do about motivation but if there are barriers that > we can lower, please let me know. For example, you could volunteer to produce OSX binaries in a timely manner for the next two years. Now would be a good point to do so, since your next release would still be a beta release. Expect to mess up the first time you do it, so it's good that it would be only a beta. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:37:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:37:14 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC5D596.9030303@gmail.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC5D596.9030303@gmail.com> Message-ID: <4BC60B5A.50501@v.loewis.de> > I seem to recall there was some issue (aside from the current lack of a > reliable OS X buildbot) that prevented us from regularly grabbing the > head of the tree and using it to automatically build the Windows and Mac > installers (to check that the installers could actually be created, > preventing a repeat of the Mac situation with 2.7b1). > > Am I misremembering the discussion from last time this topic came up? I only remember a similar discussion, which was about why the Windows nightly builds had been dropped. The reason was that the process was too flaky and required too much manual fixing, and that the demand for these binaries was too low. I don't recall OSX being mentioned in that discussion, though. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:41:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:41:39 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5EAB9.3030306@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <19397.59102.929541.18939@montanaro.dyndns.org> <4BC5EAB9.3030306@voidspace.org.uk> Message-ID: <4BC60C63.4000307@v.loewis.de> > Sure - but probably not your average Python-on-Mac user. Or at least a > good proportion of them, particularly newbies who we are keen to keep > the experience of obtaining Python simple. First download and then > install 1gigabyte of developer tools (seriously) requiring registration, > then compile Python from source yourself, is not the way to go. (And yes > the XCode tools are included in the OS disks - but not the latest > versions, especially if you are still running Leopard.) That's fine: the official binaries are (deliberately) *not* built with the latest tool chain. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:46:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:46:14 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <42735872743376525492939197280196737792-Webmail@me.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> Message-ID: <4BC60D76.9050704@v.loewis.de> > Speaking of which... I have a mac-mini that could be used for a > buildbot. How much work is needed to kickstart a buildbot, taking > into account that I'd prefer to have a buildbot with different > configure-flags that the default unix build (that is, I want to test > a framework build that is a universal binary). The configure flags are defined on the master, so if I know what they should be, I can arrange your slave to see them. You can't then change them easily. On the slave, you need to install buildbot, and create a slave configuration; it would the be good if the slave process would somehow get restarted automatically after a system reboot (I think there are recipes for that out there). > Creating the Mac installer is easy: just run > Mac/BuildScript/build-installer.py on an OSX 10.5 system where a > local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The > system should also not have fink or darwinports and a clean > /usr/local tree to avoid contaminating the build. Buildbot also supports uploading binaries to the master, which we did for the Windows builds. We could do that for OSX as well (in which case the release manager might be able to trigger a release build by just passing the right branch name in the master UI). Regards, Martin From martin at v.loewis.de Wed Apr 14 20:49:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:49:01 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <45888.1271268083@parc.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> Message-ID: <4BC60E1D.4030309@v.loewis.de> > Probably fine on your personal Mac. And the build scripts can probably > mask those out on their own. For a private Python installation, it doesn't actually hurt to have them on disk. The binaries may be linked with strange libraries, but it should work since the libraries themselves are also on disk. It's only that they would poison an installer - the binaries would work on no other machine. Regards, Martin From martin at v.loewis.de Wed Apr 14 20:53:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 20:53:52 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <20100414133936.GB25826@idyll.org> References: <4BA62546.10906@python.org> <4BC53D0D.2080802@palladion.com> <4BC54E9D.8080205@v.loewis.de> <4BC599DF.8060207@palladion.com> <20100414133936.GB25826@idyll.org> Message-ID: <4BC60F40.1050306@v.loewis.de> C. Titus Brown wrote: > If Georg, Benjamin, > Martin, or Ronald are interested, please just tell me (or Steve, or the PSF > board, or ...) what you want and I'll work on getting it funded. For me, my company provides all the infrastructure I need (tools, bandwidth, hardware, etc). I agreed, in return, to mention that support on our group's home page: http://www.dcl.hpi.uni-potsdam.de/ Regards, Martin P.S. and yes, HPI is a company, but associated with the University of Potsdam. From skip at pobox.com Wed Apr 14 20:54:22 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 13:54:22 -0500 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <45888.1271268083@parc.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> Message-ID: <19398.3934.111522.622407@montanaro.dyndns.org> Bill> In any case, they shouldn't be needed on buildbots maintained by Bill> the PSF. Sure. My question was related to humans building binary distributions though. Unless that becomes fully automated so the release manager can just push a button and have it built on and as-yet-nonexistent Mac OSX buildbot machine, somebody will have to generate that installer. Ronald says Fink, MacPorts and /usr/local are poison. If that's truly the case that's fine. It's just that it reduces the size of the potential binary installer build machines. Now that I think about it, it might not be sufficient to just hide those directories from the environment. The Python setup.py file has unconditional hard-coded references to /sw, /opt/local and /usr/local. That would probably have to change before you could use an "infected" machine to build the binary installer. (At the very least, searching /sw/include (for instance) could be suppressed if /sw/bin was not found in PATH. Similarly for /opt/local and /usr/local.) (ISTM the same might be true should people ever decide to once again build a Solaris installer. /opt/sfw is currently searched for Berkeley DB include files.) Skip From skip at pobox.com Wed Apr 14 21:02:27 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 14:02:27 -0500 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC60D76.9050704@v.loewis.de> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> Message-ID: <19398.4419.744973.715151@montanaro.dyndns.org> Martin> On the slave, you need to install buildbot, and create a slave Martin> configuration; it would the be good if the slave process would Martin> somehow get restarted automatically after a system reboot (I Martin> think there are recipes for that out there). A static IP address makes things a bit easier as well, though it's not impossible to run a buildbot using a dynamic IP address assuming you use dyndns.org or something similar to maintain the name-to-IP mapping. I ran a Mac OSX buildbot for the community buildbots for awhile but never did figure out at the time how to get it to fire up on reboot. That was a few years ago. I think today that would easily be accomplished with an @reboot crontab entry. Dunno if that was available way back then, but it's certainly supported on Mac OSX now. Skip From martin at v.loewis.de Wed Apr 14 21:03:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 14 Apr 2010 21:03:15 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <87eiiibkuf.fsf@rudin.co.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> Message-ID: <4BC61173.9040208@v.loewis.de> Paul Rudin wrote: > "Martin v. L?wis" writes: > >> The major difference in the "do it yourself" attitude is that Mac user >> get a compiler for free, as part of the operating system release, >> whereas for Windows, they have to pay for it (leaving alone VS Express >> for the moment). > > JOOI why ignore the express versions of the MS compilers? All (I think) > MS compilers are available for free in command line versions - it's the > GUI tools you pay for. Primarily out of a historical perspective: I think "we" started providing Windows binaries primarily because some people bought the appropriate license (of Visual Studio, and Wise Installer). IMO, that's a stronger reason than mere convenience. As for available tools: I'm not sure how you would build an AMD64 release just with those tools. Regards, Martin From martin at v.loewis.de Wed Apr 14 21:07:36 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:07:36 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> Message-ID: <4BC61278.7020108@v.loewis.de> Ned Deily wrote: > That *is* something that the PSF could help with. I > would be happy to help with that myself, although my time to do so will > be very limited for the next few weeks. The PSF still has a machine that was donated by Apple that once used to be a build slave. Unfortunately, that machine had hardware problems (or atleast appeared to have hardware problems). So if anybody would be interested into maintaining it, please let us know. Regards, Martin From martin at v.loewis.de Wed Apr 14 21:08:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:08:13 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <41208.1271258635@parc.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <41208.1271258635@parc.com> Message-ID: <4BC6129D.4020207@v.loewis.de> > I'd be happy to help where I can, too. All my automated testing of > UpLib (Windows, Ubuntu, Fedora, OS X) is done on Apple servers running > OS X and VirtualBox and Hudson, so I've got some experience there. Would you be interested in operating a build slave? Regards, Martin From martin at v.loewis.de Wed Apr 14 21:11:15 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:11:15 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <20100414145703.GF8041@idyll.org> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC5D596.9030303@gmail.com> <20100414145703.GF8041@idyll.org> Message-ID: <4BC61353.9010602@v.loewis.de> > Making the Windows binary build process automatic and robust is challenging > if you hate Windows (like I do). Martin also made the point that it's > been broken forever, so people don't seem to care :). ISTR Martin just > makes them manually. That's true. In particular, people had been asking that the MSI installers are code-signed, which I now also do. I wouldn't like to keep the private key for the PSF code signing certificate available to some machine that is readily accessible from the internet - so at least that part is manual, in the sense that I have to provide my password. Regards, Martin From martin at v.loewis.de Wed Apr 14 21:14:51 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:14:51 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <41372.1271259395@parc.com> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> Message-ID: <4BC6142B.7000002@v.loewis.de> > Michael Foord wrote: > >> Building the Mac installer requires volunteer time which I'm not sure >> that more hardware will fix - compiling a full build of Python for Mac >> OS X (with all the Python modules like Tkinter etc) requires expertise >> which only a few people have. > > That's nuts. Why isn't this expertise captured in the form of a script? Much of it is, but it still takes expertise to run the script. It would take even more expertise to capture the remaining pieces in the script, too, and no living person has that much expertise to write the script (perhaps there are one or two people, and they don't have the time). Regards, Martin From martin at v.loewis.de Wed Apr 14 21:16:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:16:13 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FA80.5000808@holdenweb.com> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <4BC5FA80.5000808@holdenweb.com> Message-ID: <4BC6147D.7050404@v.loewis.de> > How about as a first step the release build process include a check for > broken links before committing the web content for a new release? You'd have to convince the release manager to add a step to the release process. Given that the release process has already too many steps, he is probably skeptical wrt. such a proposal. Regards, Martin From martin at v.loewis.de Wed Apr 14 21:17:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:17:40 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <19397.58515.517331.130360@montanaro.dyndns.org> <20100414120442.5bba807e@heresy> Message-ID: <4BC614D4.9070702@v.loewis.de> >>From what I recall, the PC build process is pretty much routine (I > can't recall how much it's scripted, and how much it's manual, but > well-documented and simple, steps). I don't know what extra is needed > to build the final installer, but I'd be willing to have a go at > testing the existing process and if necessary adding some automation. I do *a lot* of manual steps, because I didn't have time to script it all. Regards, Martin From martin at v.loewis.de Wed Apr 14 21:21:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:21:40 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19398.3934.111522.622407@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> <19398.3934.111522.622407@montanaro.dyndns.org> Message-ID: <4BC615C4.7070105@v.loewis.de> > (ISTM the same might be true should people ever decide to once again build a > Solaris installer. /opt/sfw is currently searched for Berkeley DB include > files.) And rightly so (likewise for Fink). Primarily, the script is there to help people installing Python; packaging Python can be more difficult. I guess ActiveState has solved the problem (probably by not populating /opt/sfw on the Solaris machine that builds ActivePython). Regards, Martin From fuzzyman at voidspace.org.uk Wed Apr 14 21:30:13 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 21:30:13 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC607AB.4080203@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> Message-ID: <4BC617C5.7010907@voidspace.org.uk> On 14/04/2010 20:21, "Martin v. L?wis" wrote: >> Mac users definitely *do* expect installers. Building Python requires, I >> believe, the XCode development tools to be installed. Even then, >> building a full version of Python - with *all* the C extensions that are >> part of a Python release - is not a trivial task. >> > The same is true for any other operating system, though: you need to > install the compiler tool chain (sometimes, you need to buy it first), > and compiling Python with all extensions is not a trivial task. > Right - but we were discussing this in the context of barrier to entry, particularly to new users. We don't impose this requirement for Windows users though - we provide binary installers. I *know* we're a volunteer organisation (etc), but it is good for us to be aware of our process weaknesses without excusing ourselves. Unfortunately the Mac installer build script doesn't seem to run at all on Mac OS X 10.6 (at least not on my machine), but hopefully the situation is clarified so that one of us who does still have Mac OS X 10.5 will be able to build the installer in a timely manner for the next release should Ronald be too busy. All the best, Michael > Regards, > Martin > -- http://www.ironpythoninaction.com/ From simon at brunningonline.net Wed Apr 14 21:30:36 2010 From: simon at brunningonline.net (Simon Brunning) Date: Wed, 14 Apr 2010 20:30:36 +0100 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19398.4419.744973.715151@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> <19398.4419.744973.715151@montanaro.dyndns.org> Message-ID: On 14 April 2010 20:02, wrote: > I ran a Mac OSX buildbot for the community buildbots for awhile but never > did figure out at the time how to get it to fire up on reboot. ?That was a > few years ago. ?I think today that would easily be accomplished with an > @reboot crontab entry. ?Dunno if that was available way back then, but it's > certainly supported on Mac OSX now. launchd might be a better bet - . -- Cheers, Simon B. From martin at v.loewis.de Wed Apr 14 21:37:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 21:37:23 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC617C5.7010907@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> Message-ID: <4BC61973.9020207@v.loewis.de> > Right - but we were discussing this in the context of barrier to entry, > particularly to new users. We don't impose this requirement for Windows > users though - we provide binary installers. > > I *know* we're a volunteer organisation (etc), but it is good for us to > be aware of our process weaknesses without excusing ourselves. I think nobody here is questioning that it is desirable to have OSX binaries. And I readily admit that this is a weakness. > Unfortunately the Mac installer build script doesn't seem to run at all > on Mac OS X 10.6 (at least not on my machine), but hopefully the > situation is clarified so that one of us who does still have Mac OS X > 10.5 will be able to build the installer in a timely manner for the next > release should Ronald be too busy. I'm not sure whether 10.5 would be sufficient - it may be that you need to go back to 10.4 (*). Unfortunately, Apple manages to break compatibility and portability with every release, which makes this particular build task soooo tricky. You have to make all kinds of decisions and compromises where are really difficult to keep track of. Regards, Martin (*) I know that this is the version I had to go back to when I created OSX binaries for 2.5.x. From fuzzyman at voidspace.org.uk Wed Apr 14 21:41:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 21:41:46 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC61973.9020207@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> <4BC61973.9020207@v.loewis.de> Message-ID: <4BC61A7A.1070601@voidspace.org.uk> On 14/04/2010 21:37, "Martin v. L?wis" wrote: > [snip...] >> Unfortunately the Mac installer build script doesn't seem to run at all >> on Mac OS X 10.6 (at least not on my machine), but hopefully the >> situation is clarified so that one of us who does still have Mac OS X >> 10.5 will be able to build the installer in a timely manner for the next >> release should Ronald be too busy. >> > I'm not sure whether 10.5 would be sufficient - it may be that you need > to go back to 10.4 (*). Unfortunately, Apple manages to break > compatibility and portability with every release, which makes this > particular build task soooo tricky. You have to make all kinds of > decisions and compromises where are really difficult to keep track of. > > In an earlier email Ronald said: > > Creating the Mac installer is easy: just run > Mac/BuildScript/build-installer.py on an OSX 10.5 system where a local > version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The system > should also not have fink or darwinports and a clean /usr/local tree > to avoid contaminating the build. I can't verify that this is correct, I can verify it that the build-installer script doesn't appear to work under 10.6. Michael > Regards, > Martin > > (*) I know that this is the version I had to go back to when I created > OSX binaries for 2.5.x. > -- http://www.ironpythoninaction.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zvezdan at zope.com Wed Apr 14 21:44:06 2010 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Wed, 14 Apr 2010 15:44:06 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> <19398.4419.744973.715151@montanaro.dyndns.org> Message-ID: On Apr 14, 2010, at 3:30 PM, Simon Brunning wrote: > On 14 April 2010 20:02, wrote: >> I ran a Mac OSX buildbot for the community buildbots for awhile but never did figure out at the time how to get it to fire up on reboot. That was a few years ago. I think today that would easily be accomplished with an @reboot crontab entry. Dunno if that was available way back then, but it's certainly supported on Mac OSX now. > > launchd might be a better bet - . Exactly, launchd is the program intended for such use. A small nitpick at that property list shown in the linked example: It should be "Apple" now instead of "Apple Computer". They officially changed the name and all the new developer documentation shows the property lists with "-//Apple/DTD ..." :-) Regards, Zvezdan From ronaldoussoren at mac.com Wed Apr 14 22:04:47 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Apr 2010 22:04:47 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19397.65086.457847.833679@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> Message-ID: On 14 Apr, 2010, at 19:41, skip at pobox.com wrote: > > Ronald> Creating the Mac installer is easy: just run > Ronald> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a > Ronald> local version of Tcl/Tk 8.4 is installed in > Ronald> /Library/Frameworks. The system should also not have fink or > Ronald> darwinports and a clean /usr/local tree to avoid contaminating > Ronald> the build. > > For those of us who live in an X11 world and use tools which Apple doesn't > provide, like it or not, Fink or MacPorts are often a practical necessity. > Would it be sufficient to modify the environment so that /sw, /opt/local and > /usr/local don't appear in any paths when the script is run? On my laptop > (MacPorts installed, not Fink) I see that MANPATH, PATH and INFOPATH are > currently "polluted" with /opt/local. MANPATH and PATH contain /usr/local. /usr/local/lib and /usr/local/include must be empty, or at least not contain anything that might be used by the python build. AFAIK it is practically impossible[1] to tell the compiler to exclude /usr/local from its search paths. Fink and Macports must be disabled because Python's setup.py actively looks in those directory for dependencies (such as for BerkelyDB). I wouldn't mind having a configure switch that disables looking in the default Fink and Macports trees, but don't have time to work on that myself. BTW. Excluding these locations is only needed when building the official installer, you can obviously just keep them around when you build and use Python locally. I tend to move /usr/local aside when I build the installer and move it back afterwards, and have used an mostly empty virtual machine for the last couple of releases. Ronald [1] that is, the only way I've found it to tell gcc not to use any search paths at all and manually specify all search paths, including compiler-specific directories. > > Skip -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From zvezdan at zope.com Wed Apr 14 22:04:58 2010 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Wed, 14 Apr 2010 16:04:58 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC61973.9020207@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> <4BC61973.9020207@v.loewis.de> Message-ID: <35AEF1A0-5202-4925-8184-3C418596C788@zope.com> On Apr 14, 2010, at 3:37 PM, Martin v. L?wis wrote: > I'm not sure whether 10.5 would be sufficient - it may be that you need to go back to 10.4 (*). I think you just need to supply to configure MACOSX_DEPLOYMENT_TARGET=10.4 and have the appropriate SDK installed with Xcode. I believe it is installed by default with Xcode on Leopard (10.5), but with Xcode on Snow Leopard (10.6) the 10.4 SDK is an optional install. > Unfortunately, Apple manages to break compatibility and portability with every release, which makes this particular build task soooo tricky. You have to make all kinds of decisions and compromises where are really difficult to keep track of. Hmm. Apple provided compatibility SDK and documented it. The only compromise I see in this build process right now is that we are building a Panther (10.3) compatible installer, while Mac OS X is a certified UNIX starting with 10.5. So, we are deliberately using obsolete model to provide backward compatibility. However, I'm fine with that for the installer. Power users can compile their own build optimized for their platform. Regards, Zvezdan From brian.curtin at gmail.com Wed Apr 14 22:05:38 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 14 Apr 2010 15:05:38 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC61173.9040208@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> <4BC61173.9040208@v.loewis.de> Message-ID: On Wed, Apr 14, 2010 at 14:03, "Martin v. L?wis" wrote: > Paul Rudin wrote: > > "Martin v. L?wis" writes: > > > >> The major difference in the "do it yourself" attitude is that Mac user > >> get a compiler for free, as part of the operating system release, > >> whereas for Windows, they have to pay for it (leaving alone VS Express > >> for the moment). > > > > JOOI why ignore the express versions of the MS compilers? All (I think) > > MS compilers are available for free in command line versions - it's the > > GUI tools you pay for. > > Primarily out of a historical perspective: I think "we" started > providing Windows binaries primarily because some people bought the > appropriate license (of Visual Studio, and Wise Installer). IMO, that's > a stronger reason than mere convenience. > > As for available tools: I'm not sure how you would build an AMD64 > release just with those tools. > > Regards, > Martin Outside of hacking the registry and various config files, 64-bit builds can't be done with the express versions of Visual Studio, and I wouldn't be comfortable shipping a product built in that way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Wed Apr 14 22:20:19 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Apr 2010 22:20:19 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC60D76.9050704@v.loewis.de> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> Message-ID: <7CA3C28B-F6F4-4178-9895-66224F94B3BD@mac.com> On 14 Apr, 2010, at 20:46, Martin v. L?wis wrote: >> Speaking of which... I have a mac-mini that could be used for a >> buildbot. How much work is needed to kickstart a buildbot, taking >> into account that I'd prefer to have a buildbot with different >> configure-flags that the default unix build (that is, I want to test >> a framework build that is a universal binary). > > The configure flags are defined on the master, so if I know what they > should be, I can arrange your slave to see them. You can't then change > them easily. That's fine. I'll first create a stable environment for performing builds/tests and contact you when I'm ready to be added to the buildbot farm. > > On the slave, you need to install buildbot, and create a slave > configuration; it would the be good if the slave process would somehow > get restarted automatically after a system reboot (I think there are > recipes for that out there). That should be easy enough to arange. The buildbot manual mentions an @reboot crontab entry, but a launchd item should be easy enough and is slightly cleaner. > >> Creating the Mac installer is easy: just run >> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a >> local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The >> system should also not have fink or darwinports and a clean >> /usr/local tree to avoid contaminating the build. > > Buildbot also supports uploading binaries to the master, which we did > for the Windows builds. We could do that for OSX as well (in which case > the release manager might be able to trigger a release build by just > passing the right branch name in the master UI). How would that work? Creation of the OSX installer is not integrated in the regular Makefiles but is a separate script. A slightly problematic issue is that the machine I want to use for the buildbot is running OSX 10.6 and creating the binary installer doesn't work on 10.6 yet, but that should be easy enough to fix when I look into it. Ronald > > Regards, > Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From skip at pobox.com Wed Apr 14 22:23:41 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Apr 2010 15:23:41 -0500 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> <19398.4419.744973.715151@montanaro.dyndns.org> Message-ID: <19398.9293.704674.414329@montanaro.dyndns.org> >> I ran a Mac OSX buildbot for the community buildbots for awhile but >> never did figure out at the time how to get it to fire up on >> reboot. ??That was a few years ago. ??I think today that would easily >> be accomplished with an @reboot crontab entry. ??Dunno if that was >> available way back then, but it's certainly supported on Mac OSX now. Simon> launchd might be a better bet - . Sure, but it's Mac-specific. For someone like myself who moves between different platforms I generally try to use tools which work across those platforms to the extent that I can. While @reboot is not universally available (Solaris doesn't support it as far as I can tell) it is still more common where I operate than launchd. Not to mention which I have to horse around with XML for config files which I completely detest for that purpose. Skip From brett at python.org Wed Apr 14 22:32:31 2010 From: brett at python.org (Brett Cannon) Date: Wed, 14 Apr 2010 13:32:31 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: On Tue, Apr 13, 2010 at 20:54, Chris Jerdonek wrote: > Here is another patch for review: > > http://bugs.python.org/issue8370 > > This is a trivial fix to the 2.6 and 2.7 documentation. > > There is no need to email python-dev about individual patches just to get them looked at. There is a mailing list that we all subscribe to that send an email on all new issues and another one on every change to any issue. You should only email python-dev if a patch you wrote has been sitting around for a very long time and is not being actively looked at or you think it should hold up a release. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Apr 14 22:37:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 22:37:04 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <7CA3C28B-F6F4-4178-9895-66224F94B3BD@mac.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC60D76.9050704@v.loewis.de> <7CA3C28B-F6F4-4178-9895-66224F94B3BD@mac.com> Message-ID: <4BC62770.2010107@v.loewis.de> > How would that work? Creation of the OSX installer is not integrated > in the regular Makefiles but is a separate script. Again by setting up another builder (and assigning it to either the same or a different slave). Instead of the regular configure/make/make test/make clean builder, this one would configure/run build script/upload/clean. Essentially, a builder will invoke arbitrary shell scripts on the slave, which will (slavishly :-) perform them. So make sure this can run under an untrusted account; anybody with control over the master can get access to all slaves (but we do keep access to the master very tightly restricted). In addition, all committers basically have control over all slaves (by committing stuff into the makefile); this is, of course, monitored through the checkins list. I recommend we look at daily installers after the regular slave is set up. > A slightly problematic issue is that the machine I want to use for > the buildbot is running OSX 10.6 and creating the binary installer > doesn't work on 10.6 yet, but that should be easy enough to fix when > I look into it. So we should definitely take one step at a time. Regards, Martin From martin at v.loewis.de Wed Apr 14 22:40:19 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Apr 2010 22:40:19 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <35AEF1A0-5202-4925-8184-3C418596C788@zope.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> <4BC61973.9020207@v.loewis.de> <35AEF1A0-5202-4925-8184-3C418596C788@zope.com> Message-ID: <4BC62833.4010609@v.loewis.de> > I think you just need to supply to configure > > MACOSX_DEPLOYMENT_TARGET=10.4 > > and have the appropriate SDK installed with Xcode. Wouldn't that break 10.3 compatibility (seel below)? >> Unfortunately, Apple manages to break compatibility and portability >> with every release, which makes this particular build task soooo >> tricky. You have to make all kinds of decisions and compromises >> where are really difficult to keep track of. > > > Hmm. Apple provided compatibility SDK and documented it. > > The only compromise I see in this build process right now is that we > are building a Panther (10.3) compatible installer, while Mac OS X is > a certified UNIX starting with 10.5. I think there are more issues. People want a fat binary that supports AMD64 along with x86, yet building such a binary requires an SDK that won't support PPC anymore - right? Regards, Martin From chris.jerdonek at gmail.com Wed Apr 14 22:41:39 2010 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 14 Apr 2010 13:41:39 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: On Wed, Apr 14, 2010 at 1:32 PM, Brett Cannon wrote: > There is no need to email python-dev about individual patches just to get > them looked at. There is a mailing list that we all subscribe to that send > an email on all new issues and another one on every change to any issue. You > should only email python-dev if a patch you wrote has been sitting around > for a very long time and is not being actively looked at or you think it > should hold up a release. Sorry, I had received somewhat different guidance on tracker-discuss: http://mail.python.org/pipermail/tracker-discuss/2010-April/002482.html http://mail.python.org/pipermail/tracker-discuss/2010-April/002483.html http://mail.python.org/pipermail/tracker-discuss/2010-April/002484.html Otherwise, I would not have bothered to e-mail the list. I will be more conservative about posting to python-dev in the future. --Chris From rdmurray at bitdance.com Wed Apr 14 23:07:27 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 14 Apr 2010 17:07:27 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <742c849c-6fe9-435a-9cbe-aaba084fd654@email.android.com> References: =?US-ASCII?Q?=3C4BA62546=2E10906=40python=2Eorg=3E=0D=0A=09=3Chq3?= =?US-ASCII?Q?e52=248oj=241=40dough=2Egmane=2Eorg=3E=09=3C4BC54E6B?= =?US-ASCII?Q?=2E3090209=40v=2Eloewis=2Ede=3E=0D=0A=09=3C4BC54FD3=2E4?= =?US-ASCII?Q?030907=40holdenweb=2Ecom=3E=09=3C4BC55459=2E60009?= =?US-ASCII?Q?04=40v=2Eloewis=2Ede=3E=0D=0A=09=3C87eiiibkuf=2Efsf=40?= =?US-ASCII?Q?rudin=2Eco=2Euk=3E=0D=0A=09=3Cm2p79990c6b1004140359?= =?US-ASCII?Q?ze789ae7n6d7a2bffba5bd9be=40mail=2Egmail?= =?US-ASCII?Q?=2Ecom=3E_=3C4BC5FD04=2E8060208=40holdenweb=2Ecom=3E?= <742c849c-6fe9-435a-9cbe-aaba084fd654@email.android.com> Message-ID: <20100414210727.9DFD01AB33E@kimball.webabinitio.net> On Wed, 14 Apr 2010 14:06:44 -0400, Eric Smith wrote: > "Steve Holden" wrote: >> I spent some considerable effort last year ensuring the developer >> community was well-supplied with MS developer licenses that give access >> to any necessary tools. Was I wasting my time? > > In my case it was not a waste of time. I use MSDN for dev and testing. Just not > for release building. Ditto. Without the license I wouldn't have a Windows machine that I could run tests on at all (I run the OS in KVM instances). -- R. David Murray www.bitdance.com From janssen at parc.com Wed Apr 14 23:10:01 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 14:10:01 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC6129D.4020207@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <41208.1271258635@parc.com> <4BC6129D.4020207@v.loewis.de> Message-ID: <49527.1271279401@parc.com> Martin v. L?wis wrote: > > I'd be happy to help where I can, too. All my automated testing of > > UpLib (Windows, Ubuntu, Fedora, OS X) is done on Apple servers running > > OS X and VirtualBox and Hudson, so I've got some experience there. > > Would you be interested in operating a build slave? I could help to maintain some OS X buildbots located somewhere, sure. Bill From janssen at parc.com Wed Apr 14 23:16:17 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 14:16:17 PDT Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC6142B.7000002@v.loewis.de> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> <4BC6142B.7000002@v.loewis.de> Message-ID: <49589.1271279777@parc.com> Martin v. L?wis wrote: > > Michael Foord wrote: > > > >> Building the Mac installer requires volunteer time which I'm not sure > >> that more hardware will fix - compiling a full build of Python for Mac > >> OS X (with all the Python modules like Tkinter etc) requires expertise > >> which only a few people have. > > > > That's nuts. Why isn't this expertise captured in the form of a script? > > Much of it is, but it still takes expertise to run the script. Apparently not, according to Ronald's recent mail. I'd be happy to help with whatever remaining capture is necessary for OS X -- not automating this kind of thing is nuts. My buildbots build UpLib, and most of its optional packages (including ghostscript, xpdf, libtiff, openssl, t1lib, Lucene, PyLucene, etc.) from source every night, and whenever I do an UpLib check-in. On Linux and OS X, right now -- I'm working on Windows right now. > It would take even more expertise to capture the remaining pieces in the > script, too, and no living person has that much expertise to write the > script (perhaps there are one or two people, and they don't have the time). Well, God forbid they should ever be hit by a bus! This kind of thing needs to be written down. Bill From brett at python.org Wed Apr 14 23:19:02 2010 From: brett at python.org (Brett Cannon) Date: Wed, 14 Apr 2010 14:19:02 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: On Wed, Apr 14, 2010 at 13:41, Chris Jerdonek wrote: > On Wed, Apr 14, 2010 at 1:32 PM, Brett Cannon wrote: > > There is no need to email python-dev about individual patches just to get > > them looked at. There is a mailing list that we all subscribe to that > send > > an email on all new issues and another one on every change to any issue. > You > > should only email python-dev if a patch you wrote has been sitting around > > for a very long time and is not being actively looked at or you think it > > should hold up a release. > > Sorry, I had received somewhat different guidance on tracker-discuss: > > http://mail.python.org/pipermail/tracker-discuss/2010-April/002482.html > http://mail.python.org/pipermail/tracker-discuss/2010-April/002483.html > http://mail.python.org/pipermail/tracker-discuss/2010-April/002484.html > > Otherwise, I would not have bothered to e-mail the list. > I see the confusion. I think Martin meant more about open issues that required discussion, not simply issues that had a patch ready to go. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Apr 14 23:32:07 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Apr 2010 09:32:07 +1200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC586BA.9010704@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> Message-ID: <4BC63457.5020006@canterbury.ac.nz> Michael Foord wrote: > Building Python requires, I > believe, the XCode development tools to be installed. Even then, > building a full version of Python - with *all* the C extensions that are > part of a Python release - is not a trivial task. What's non-trivial about it? I usually find that the normal "./configure; make; make install" sequence works fine without any further intervention. If you want a framework installation you have to read the README and use a couple of extra options, but it still works very smoothly. -- Greg From brett at python.org Wed Apr 14 23:12:51 2010 From: brett at python.org (Brett Cannon) Date: Wed, 14 Apr 2010 14:12:51 -0700 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100413162107.171c724c@heresy> References: <20100413162107.171c724c@heresy> Message-ID: On Tue, Apr 13, 2010 at 13:21, Barry Warsaw wrote: > I am attaching the latest revision of PEP 3147 to this message, which is > also > available here: > > http://www.python.org/dev/peps/pep-3147/ > > I think the PEP is ready for pronouncement, and the patch is pretty much > ready > for merging into py3k. The only thing that I can think of that is not > implemented yet is this section on PEP 302 loaders: > > PEP 302 [18]_ defined loaders have a `.get_filename()` method which > points to the `__file__` for a module. As part of this PEP, we will > extend this API, to include a new method `.get_paths()` which will > return a 2-tuple containing the path to the source file and the path > to where the matching `pyc` file is (or would be). > > I'm honestly not sure whether this is still essential, or whether the > importlib ABC changes Brett and I talked about at Pycon are still required. > I > now believe they are at best a minor part of the implementation if so. > Maybe > Brett can chime in on that. > As long as the importlib ABCs can always set __cached__ to None or blindly use imp.source_to_cache() then there is no backwards-compatibility issue for importlib that I can think of requiring get_paths(). I say leave get_paths() out of the PEP and just let importlib handle any API changes that are needed for itself. Otherwise loaders that just follow PEP 302 can simply set __cached__ to None and not worry about things. Otherwise Nick is the only person who might have an immediate need for get_paths(), but I suspect that as long as get_filename() works his end is covered as well. And just a quick suggestion: can we standardize what imp.source_to_path() and friend are supposed to return if the interpreter doesn't support bytecode? I will probably have to rely on that for something so it would be best to say now whether it should be None or raise an exception so there is no divergence on this between VMs. -Brett > > Everything else is implemented, tested, and has undergone four rounds of > Rietveld reviews (thanks Antoine, Benjamin, Brett, and Georg!). A fifth > patch > set has been uploaded and is available here: > > http://codereview.appspot.com/842043/show > > This addresses all previous comments, includes some fixes from Brian Curtin > for Windows (thanks!) and fixes __main__ and -m support. I'd like to > commit > this to py3k sooner rather than later so that we can shake out any > additional > issues that might crop up, without having to continue to maintain my > external > branches. > > Guido, what say you? > -Barry > > PEP: 3147 > Title: PYC Repository Directories > Version: $Revision: 80025 $ > Last-Modified: $Date: 2010-04-12 22:17:40 -0400 (Mon, 12 Apr 2010) $ > Author: Barry Warsaw > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2009-12-16 > Python-Version: 3.2 > Post-History: 2010-01-30, 2010-02-25, 2010-03-03, 2010-04-12 > > > Abstract > ======== > > This PEP describes an extension to Python's import mechanism which > improves sharing of Python source code files among multiple installed > different versions of the Python interpreter. It does this by > allowing more than one byte compilation file (.pyc files) to be > co-located with the Python source file (.py file). The extension > described here can also be used to support different Python > compilation caches, such as JIT output that may be produced by an > Unladen Swallow [1]_ enabled C Python. > > > Background > ========== > > CPython compiles its source code into "byte code", and for performance > reasons, it caches this byte code on the file system whenever the > source file has changes. This makes loading of Python modules much > faster because the compilation phase can be bypassed. When your > source file is `foo.py`, CPython caches the byte code in a `foo.pyc` > file right next to the source. > > Byte code files contain two 32-bit numbers followed by the marshaled > [2]_ code object. The 32-bit numbers represent a magic number and a > timestamp. The magic number changes whenever Python changes the byte > code format, e.g. by adding new byte codes to its virtual machine. > This ensures that pyc files built for previous versions of the VM > won't cause problems. The timestamp is used to make sure that the pyc > file is not older than the py file that was used to create it. When > either the magic number or timestamp do not match, the py file is > recompiled and a new pyc file is written. > > In practice, it is well known that pyc files are not compatible across > Python major releases. A reading of import.c [3]_ in the Python > source code proves that within recent memory, every new CPython major > release has bumped the pyc magic number. > > > Rationale > ========= > > Linux distributions such as Ubuntu [4]_ and Debian [5]_ provide more > than one Python version at the same time to their users. For example, > Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, > with Python 2.6 being the default. > > This causes a conflict for Python source files installed by the > system (including third party packages), because you cannot compile a > single Python source file for more than one Python version at a time. > Thus if your system wanted to install a `/usr/share/python/foo.py`, it > could not create a `/usr/share/python/foo.pyc` file usable across all > installed Python versions. > > Furthermore, in order to ease the burden on operating system packagers > for these distributions, the distribution packages do not contain > Python version numbers [6]_; they are shared across all Python > versions installed on the system. Putting Python version numbers in > the packages would be a maintenance nightmare, since all the packages > - *and their dependencies* - would have to be updated every time a new > Python release was added or removed from the distribution. Because of > the sheer number of packages available, this amount of work is > infeasible. > > C extensions can be source compatible across multiple versions of > Python. Compiled extension modules are usually not compatible though, > and PEP 384 [7]_ has been proposed to address this by defining a > stable ABI for extension modules. > > Because these distributions cannot share pyc files, elaborate > mechanisms have been developed to put the resulting pyc files in > non-shared locations while the source code is still shared. Examples > include the symlink-based Debian regimes python-support [8]_ and > python-central [9]_. These approaches make for much more complicated, > fragile, inscrutable, and fragmented policies for delivering Python > applications to a wide range of users. Arguably more users get Python > from their operating system vendor than from upstream tarballs. Thus, > solving this pyc sharing problem for CPython is a high priority for > such vendors. > > This PEP proposes a solution to this problem. > > > Proposal > ======== > > Python's import machinery is extended to write and search for byte > code cache files in a single directory inside every Python package > directory. This directory will be called `__pycache__`. > > Further, pyc files will contain a magic string that differentiates the > Python version they were compiled for. This allows multiple byte > compiled cache files to co-exist for a single Python source file. > > This scheme has the added benefit of reducing the clutter in a Python > package directory. > > When a Python source file is imported for the first time, a > `__pycache__` directory will be created in the package directory, if > one does not already exist. The pyc file for the imported source will > be written to the `__pycache__` directory, using the magic-tag > formatted name. If either the creation of the `__pycache__` directory > or the pyc file inside that fails, the import will still succeed, just > as it does in a pre-PEP-3147 world. > > If the py source file is missing, the pyc file inside `__pycache__` > will be ignored. This eliminates the problem of accidental stale pyc > file imports. > > For backward compatibility, Python will still support pyc-only > distributions, however it will only do so when the pyc file lives in > the directory where the py file *would* have been, i.e. not in the > `__pycache__` directory. pyc file outside of `__pycache__` will only > be imported if the py source file is missing. > > Tools such as `py_compile` [15]_ and `compileall` [16]_ will be > extended to create PEP 3147 formatted layouts automatically, but will > have an option to create pyc-only distribution layouts. > > > Examples > --------- > > What would this look like in practice? > > Let's say we have a Python package named `alpha` which contains a > sub-package name `beta`. The source directory layout before byte > compilation might look like this:: > > alpha/ > __init__.py > one.py > two.py > beta/ > __init__.py > three.py > four.py > > After byte compiling this package with Python 3.2, you would see the > following layout:: > > alpha/ > __pycache__/ > __init__.cpython-32.pyc > one.cpython-32.pyc > two.cpython-32.pyc > __init__.py > one.py > two.py > beta/ > __pycache__/ > __init__.cpython-32.pyc > three.cpython-32.pyc > four.cpython-32.pyc > __init__.py > three.py > four.py > > *Note: listing order may differ depending on the platform.* > > Let's say that two new versions of Python are installed, one is Python > 3.3 and another is Unladen Swallow. After byte compilation, the file > system would look like this:: > > alpha/ > __pycache__/ > __init__.cpython-32.pyc > __init__.cpython-33.pyc > __init__.unladen-10.pyc > one.cpython-32.pyc > one.cpython-33.pyc > one.unladen-10.pyc > two.cpython-32.pyc > two.cpython-33.pyc > two.unladen-10.pyc > __init__.py > one.py > two.py > beta/ > __pycache__/ > __init__.cpython-32.pyc > __init__.cpython-33.pyc > __init__.unladen-10.pyc > three.cpython-32.pyc > three.cpython-33.pyc > three.unladen-10.pyc > four.cpython-32.pyc > four.cpython-33.pyc > four.unladen-10.pyc > __init__.py > three.py > four.py > > As you can see, as long as the Python version identifier string is > unique, any number of pyc files can co-exist. These identifier > strings are described in more detail below. > > A nice property of this layout is that the `__pycache__` directories > can generally be ignored, such that a normal directory listing would > show something like this:: > > alpha/ > __pycache__/ > __init__.py > one.py > two.py > beta/ > __pycache__/ > __init__.py > three.py > four.py > > This is much less cluttered than even today's Python. > > > Python behavior > =============== > > When Python searches for a module to import (say `foo`), it may find > one of several situations. As per current Python rules, the term > "matching pyc" means that the magic number matches the current > interpreter's magic number, and the source file is not newer than the > `pyc` file. > > > Case 1: The first import > ------------------------ > > When Python is asked to import module `foo`, it searches for a > `foo.py` file (or `foo` package, but that's not important for this > discussion) along its `sys.path`. When Python locates the `foo.py` > file it will look for a `__pycache__` directory in the directory where > it found the `foo.py`. If the `__pycache__` directory is missing, > Python will create it. Then it will parse and byte compile the > `foo.py` file and save the byte code in `__pycache__/foo..pyc`, > where is defined by the Python implementation, but will be a > human readable string such as `cpython-32`. > > > Case 2: The second import > ------------------------- > > When Python is asked to import module `foo` a second time (in a > different process of course), it will again search for the `foo.py` > file along its `sys.path`. When Python locates the `foo.py` file, it > looks for a matching `__pycache__/foo..pyc` and finding this, > it reads the byte code and continues as usual. > > > Case 3: __pycache__/foo..pyc with no source > --------------------------------------------------- > > It's possible that the `foo.py` file somehow got removed, while > leaving the cached pyc file still on the file system. If the > `__pycache__/foo..pyc` file exists, but the `foo.py` file used > to create it does not, Python will raise an `ImportError` when asked > to import foo. In other words, Python will not import a pyc file from > the cache directory unless the source file exists. > > > Case 4: legacy pyc files and source-less imports > ------------------------------------------------ > > Python will ignore all legacy pyc files when a source file exists next > to it. In other words, if a `foo.pyc` file exists next to the > `foo.py` file, the pyc file will be ignored in all cases > > In order to continue to support source-less distributions though, if > the source file is missing, Python will import a lone pyc file if it > lives where the source file would have been. > > > Case 5: read-only file systems > ------------------------------ > > When the source lives on a read-only file system, or the `__pycache__` > directory or pyc file cannot otherwise be written, all the same rules > apply. This is also the case when `__pycache__` happens to be written > with permissions which do not allow for writing containing pyc files. > > > > Flow chart > ========== > > Here is a flow chart describing how modules are loaded: > > .. image:: pep-3147-1.png > :scale: 75 > > > Magic identifiers > ================= > > pyc files inside of the `__pycache__` directories contain a magic > identifier in their file names. These are mnemonic tags for the > actual magic numbers used by the importer. For example, in Python > 3.2, we could use the hexlified [10]_ magic number as a unique > identifier:: > > >>> from binascii import hexlify > >>> from imp import get_magic > >>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii')) > 'foo.580c0d0a.pyc' > > This isn't particularly human friendly though. Instead, this PEP > proposes a *magic tag* that uniquely defines `.pyc` files for the > current version of Python. Whenever the magic number is bumped, a new > magic tag is defined which is unique among all versions and > implementations of Python. The actual contents of the magic tag is > left up to the implementation, although it is recommended that the tag > include the implementation name and a version shorthand. In general, > magic numbers never change between Python micro releases, but the > convention can be extended to handle magic number changes between > pre-release development versions. > > For example, CPython 3.2 would have a magic tag of `cpython-32` and > write pyc files like this: `foo.cpython-32.pyc`. When the `-O` flag > is used, it would write `foo.cpython-32.pyo`. For backports of this > feature to Python 2, when the `-U` flag is used, a file such as > `foo.cpython-27u.pyc` can be written. > > The magic tag is available in the `imp` module via the `get_tag()` > function. This is analogous to the `get_magic()` function already > available in that module. > > > Alternative Python implementations > ================================== > > Alternative Python implementations such as Jython [11]_, IronPython > [12]_, PyPy [13]_, Pynie [14]_, and Unladen Swallow can also use the > `__pycache__` directory to store whatever compilation artifacts make > sense for their platforms. For example, Jython could store the class > file for the module in `__pycache__/foo.jython-32.class`. > > > Implementation strategy > ======================= > > This feature is targeted for Python 3.2, solving the problem for those > and all future versions. It may be back-ported to Python 2.7. > Vendors are free to backport the changes to earlier distributions as > they see fit. > > > Effects on existing code > ======================== > > Adoption of this PEP will affect existing code and idioms, both inside > Python and outside. This section enumerates some of these effects. > > > __file__ > --------- > > In Python 3, when you import a module, its `__file__` attribute points > to its source `py` file (in Python 2, it points to the `pyc` file). A > package's `__file__` points to the `py` file for its `__init__.py`. > E.g.:: > > >>> import foo > >>> foo.__file__ > 'foo.py' > # baz is a package > >>> import baz > >>> baz.__file__ > 'baz/__init__.py' > > Nothing in this PEP would change the semantics of `__file__`. > > This PEP proposes the addition of an `__cached__` attribute to > modules, which will always point to the actual `pyc` file that was > read or written. When the environment variable > `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if > the source lives on a read-only filesystem, then the `__cached__` > attribute will point to the location that the `pyc` file *would* have > been written to if it didn't exist. This location of course includes > the `__pycache__` subdirectory in its path. > > For alternative Python implementations which do not support `pyc` > files, the `__cached__` attribute may point to whatever information > makes sense. E.g. on Jython, this might be the `.class` file for the > module: `__pycache__/foo.jython-32.class`. Some implementations may > use multiple compiled files to create the module, in which case > `__cached__` may be a tuple. The exact contents of `__cached__` are > Python implementation specific. > > It is recommended that when nothing sensible can be calculated, > implementations should set the `__cached__` attribute to `None`. > > > py_compile and compileall > ------------------------- > > Python comes with two modules, `py_compile` [15]_ and `compileall` > [16]_ which support compiling Python modules external to the built-in > import machinery. `py_compile` in particular has intimate knowledge > of byte compilation, so these will be updated to understand the new > layout. The `-b` flag is added to `compileall` for writing legacy > `.pyc` byte-compiled file path names. > > > bdist_wininst and the Windows installer > --------------------------------------- > > These tools also compile modules explicitly on installation. If they > do not use `py_compile` and `compileall`, then they would also have to > be modified to understand the new layout. > > > File extension checks > --------------------- > > There exists some code which checks for files ending in `.pyc` and > simply chops off the last character to find the matching `.py` file. > This code will obviously fail once this PEP is implemented. > > To support this use case, we'll add two new methods to the `imp` > package [17]_: > > * `imp.source_from_cache(py_path)` -> `pyc_path` > * `imp.cache_from_source(pyc_path)` -> `py_path` > > Alternative implementations are free to override these functions to > return reasonable values based on their own support for this PEP. > > > PEP 302 loaders > --------------- > > PEP 302 [18]_ defined loaders have a `.get_filename()` method which > points to the `__file__` for a module. As part of this PEP, we will > extend this API, to include a new method `.get_paths()` which will > return a 2-tuple containing the path to the source file and the path > to where the matching `pyc` file is (or would be). > > > Backports > --------- > > For versions of Python earlier than 3.2 (and possibly 2.7), it is > possible to backport this PEP. However, in Python 3.2 (and possibly > 2.7), this behavior will be turned on by default, and in fact, it will > replace the old behavior. Backports will need to support the old > layout by default. We suggest supporting PEP 3147 through the use of > an environment variable called `$PYTHONENABLECACHEDIR` or the command > line switch `-Xenablecachedir` to enable the feature. > > > Makefiles and other dependency tools > ------------------------------------ > > Makefiles and other tools which calculate dependencies on `.pyc` files > (e.g. to byte-compile the source if the `.pyc` is missing) will have > to be updated to check the new paths. > > > Alternatives > ============ > > PEP 304 > ------- > > There is some overlap between the goals of this PEP and PEP 304 [19]_, > which has been withdrawn. However PEP 304 would allow a user to > create a shadow file system hierarchy in which to store `pyc` files. > This concept of a shadow hierarchy for `pyc` files could be used to > satisfy the aims of this PEP. Although the PEP 304 does not indicate > why it was withdrawn, shadow directories have a number of problems. > The location of the shadow `pyc` files would not be easily discovered > and would depend on the proper and consistent use of the > `$PYTHONBYTECODE` environment variable both by the system and by end > users. There are also global implications, meaning that while the > system might want to shadow `pyc` files, users might not want to, but > the PEP defines only an all-or-nothing approach. > > As an example of the problem, a common (though fragile) Python idiom > for locating data files is to do something like this:: > > from os import dirname, join > import foo.bar > data_file = join(dirname(foo.bar.__file__), 'my.dat') > > This would be problematic since `foo.bar.__file__` will give the > location of the `pyc` file in the shadow directory, and it may not be > possible to find the `my.dat` file relative to the source directory > from there. > > > Fat byte compilation files > -------------------------- > > An earlier version of this PEP described "fat" Python byte code files. > These files would contain the equivalent of multiple `pyc` files in a > single `pyf` file, with a lookup table keyed off the appropriate magic > number. This was an extensible file format so that the first 5 > parallel Python implementations could be supported fairly efficiently, > but with extension lookup tables available to scale `pyf` byte code > objects as large as necessary. > > The fat byte compilation files were fairly complex, and inherently > introduced difficult race conditions, so the current simplification of > using directories was suggested. The same problem applies to using > zip files as the fat pyc file format. > > > Multiple file extensions > ------------------------ > > The PEP author also considered an approach where multiple thin byte > compiled files lived in the same place, but used different file > extensions to designate the Python version. E.g. foo.pyc25, > foo.pyc26, foo.pyc31 etc. This was rejected because of the clutter > involved in writing so many different files. The multiple extension > approach makes it more difficult (and an ongoing task) to update any > tools that are dependent on the file extension. > > > .pyc > ---- > > A proposal was floated to call the `__pycache__` directory `.pyc` or > some other dot-file name. This would have the effect on *nix systems > of hiding the directory. There are many reasons why this was > rejected by the BDFL [20]_ including the fact that dot-files are only > special on some platforms, and we actually do *not* want to hide these > completely from users. > > > Reference implementation > ======================== > > Work on this code is tracked in a Bazaar branch on Launchpad [22]_ > until it's ready for merge into Python 3.2. The work-in-progress diff > can also be viewed [23]_ and is updated automatically as new changes > are uploaded. > > A Rietveld code review issue [24]_ has been opened as of 2010-04-01 (no, > this is not an April Fools joke :). > > > References > ========== > > .. [1] PEP 3146 > > .. [2] The marshal module: > http://www.python.org/doc/current/library/marshal.html > > .. [3] import.c: > > http://svn.python.org/view/python/branches/py3k/Python/import.c?view=markup > > .. [4] Ubuntu: > > .. [5] Debian: > > .. [6] Debian Python Policy: > http://www.debian.org/doc/packaging-manuals/python-policy/ > > .. [7] PEP 384 > > .. [8] python-support: > http://wiki.debian.org/DebianPythonFAQ#Whatispython-support.3F > > .. [9] python-central: > http://wiki.debian.org/DebianPythonFAQ#Whatispython-central.3F > > .. [10] binascii.hexlify(): > http://www.python.org/doc/current/library/binascii.html#binascii.hexlify > > .. [11] Jython: http://www.jython.org/ > > .. [12] IronPython: http://ironpython.net/ > > .. [13] PyPy: http://codespeak.net/pypy/dist/pypy/doc/ > > .. [14] Pynie: http://code.google.com/p/pynie/ > > .. [15] py_compile: http://docs.python.org/library/py_compile.html > > .. [16] compileall: http://docs.python.org/library/compileall.html > > .. [17] imp: http://www.python.org/doc/current/library/imp.html > > .. [18] PEP 302 > > .. [19] PEP 304 > > .. [20] http://www.mail-archive.com/python-dev at python.org/msg45203.html > > .. [21] importlib: http://docs.python.org/3.1/library/importlib.html > > .. [22] https://code.launchpad.net/~barry/python/pep3147 > > .. [23] https://code.launchpad.net/~barry/python/pep3147/+merge/22648 > > .. [24] http://codereview.appspot.com/842043/show > > > ACKNOWLEDGMENTS > =============== > > Barry Warsaw's original idea was for fat Python byte code files. > Martin von Loewis reviewed an early draft of the PEP and suggested the > simplification to store traditional `pyc` and `pyo` files in a > directory. Many other people reviewed early versions of this PEP and > provided useful feedback including but not limited to: > > * David Malcolm > * Josselin Mouette > * Matthias Klose > * Michael Hudson > * Michael Vogt > * Piotr O?arowski > * Scott Kitterman > * Toshio Kuratomi > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Wed Apr 14 23:37:29 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Apr 2010 23:37:29 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC63457.5020006@canterbury.ac.nz> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> Message-ID: <4BC63599.5020005@voidspace.org.uk> On 14/04/2010 23:32, Greg Ewing wrote: > Michael Foord wrote: >> Building Python requires, I believe, the XCode development tools to >> be installed. Even then, building a full version of Python - with >> *all* the C extensions that are part of a Python release - is not a >> trivial task. > > What's non-trivial about it? I usually find that the normal > "./configure; make; make install" sequence works fine without > any further intervention. > > If you want a framework installation you have to read the > README and use a couple of extra options, but it still works > very smoothly. > A build on my machine produces output similar to: Python build finished, but the necessary bits to build these modules were not found: _bsddb dl gdbm imageop linuxaudiodev ossaudiodev readline spwd sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules: _tkinter Obviously many of those are not meant to be built and I usually build Python for running the test suite - so I don't care about not having Tkinter. A new user of Python would most certainly care about not having Tkinter. Michael -- http://www.ironpythoninaction.com/ From paul at rudin.co.uk Wed Apr 14 23:51:17 2010 From: paul at rudin.co.uk (Paul Rudin) Date: Wed, 14 Apr 2010 22:51:17 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> Message-ID: <8739yxaeju.fsf@rudin.co.uk> Paul Moore writes: > On 14 April 2010 07:37, Paul Rudin wrote: >> "Martin v. L?wis" writes: >> >>> The major difference in the "do it yourself" attitude is that Mac user >>> get a compiler for free, as part of the operating system release, >>> whereas for Windows, they have to pay for it (leaving alone VS Express >>> for the moment). >> >> JOOI why ignore the express versions of the MS compilers? All (I think) >> MS compilers are available for free in command line versions - it's the >> GUI tools you pay for. > > I believe that the express editions don't include some of the advanced > optimisations (profile guided optimisation rings a bell) which are > used to build the official binaries. So if the binaries were built > using Express Edition, they would be somewhat slower. > > That is just my recollection, however - it may be out of date or wrong. > Paul. I could be wrong too. However I think I had two things confused (it's been a while since I've done any windows development). There are two different suites of free compilers from MS. There's the windows sdk, which includes a command line toolchain and the express edition of VS which (essentially) is a cut down version of the commercial version of VS. Some info about the windows sdk versions is here and about versions - including Express - here (at least for VC++) although that's a little old. From ncoghlan at gmail.com Thu Apr 15 00:02:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 08:02:05 +1000 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BC60B5A.50501@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC5D596.9030303@gmail.com> <4BC60B5A.50501@v.loewis.de> Message-ID: <4BC63B5D.8030400@gmail.com> Martin v. L?wis wrote: >> I seem to recall there was some issue (aside from the current lack of a >> reliable OS X buildbot) that prevented us from regularly grabbing the >> head of the tree and using it to automatically build the Windows and Mac >> installers (to check that the installers could actually be created, >> preventing a repeat of the Mac situation with 2.7b1). >> >> Am I misremembering the discussion from last time this topic came up? > > I only remember a similar discussion, which was about why the Windows > nightly builds had been dropped. The reason was that the process was too > flaky and required too much manual fixing, and that the demand for these > binaries was too low. I don't recall OSX being mentioned in that > discussion, though. Yep, that sounds like the discussion I was thinking of. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Apr 15 00:11:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 08:11:17 +1000 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC607AB.4080203@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> Message-ID: <4BC63D85.2020600@gmail.com> Martin v. L?wis wrote: > The same is true for any other operating system, though: you need to > install the compiler tool chain (sometimes, you need to buy it first), > and compiling Python with all extensions is not a trivial task. Even on Linux, it takes a bit of fiddling. I finally remembered to write the steps down for Kubuntu when I set up my current machine (you need to apt-get half a dozen or so additional dev packages): http://boredomandlaziness.blogspot.com/2010/01/kubuntu-dev-packages-to-build-python.html Take away the convenience of apt-get and add installing the compiler and installer toolchains and you've got a fair amount of setup time on your hands (and lots of things that can go wrong). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Thu Apr 15 00:11:54 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Apr 2010 23:11:54 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC5FD04.8060208@holdenweb.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC55459.6000904@v.loewis.de> <87eiiibkuf.fsf@rudin.co.uk> <4BC5FD04.8060208@holdenweb.com> Message-ID: On 14 April 2010 18:36, Steve Holden wrote: > I spent some considerable effort last year ensuring the developer > community was well-supplied with MS developer licenses that give access > to any necessary tools. Was I wasting my time? Definitely not - my offer is at least in part based on the fact that I have the full tools as a result and so can do (hopefully) useful work on assisting with any necessary build process improvements. My comment was in response to the question "why are we ignoring the express editions"? If having the full versions wasn't better than having the free ones, the developer licenses would be of less benefit (and that isn't the case, as I say). Paul. From zvezdan at zope.com Thu Apr 15 00:12:07 2010 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Wed, 14 Apr 2010 18:12:07 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC62833.4010609@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> <4BC61973.9020207@v.loewis.de> <35AEF1A0-5202-4925-8184-3C418596C788@zope.com> <4BC62833.4010609@v.loewis.de> Message-ID: <9C0AE3C0-D860-4501-8B4C-47C4614580A9@zope.com> On Apr 14, 2010, at 4:40 PM, Martin v. L?wis wrote: >> I think you just need to supply to configure >> >> MACOSX_DEPLOYMENT_TARGET=10.4 >> >> and have the appropriate SDK installed with Xcode. > > Wouldn't that break 10.3 compatibility (seel below)? I was replying to your point about 10.4 build. Naturally, if you want a 10.3 build you'd pass 10.3 as the target and would have to have appropriate Xcode SDK installed. >>> Unfortunately, Apple manages to break compatibility and portability >>> with every release, which makes this particular build task soooo >>> tricky. You have to make all kinds of decisions and compromises >>> where are really difficult to keep track of. >> >> >> Hmm. Apple provided compatibility SDK and documented it. >> >> The only compromise I see in this build process right now is that we >> are building a Panther (10.3) compatible installer, while Mac OS X is >> a certified UNIX starting with 10.5. > > I think there are more issues. People want a fat binary that supports > AMD64 along with x86, yet building such a binary requires an SDK that > won't support PPC anymore - right? Yes. x86_64, i386, and ppc are supported even in the Xcode supplied with the latest Mac OS X 10.6. Only ppc64 is not. So, ppc is not an issue. The problem is that enforcing backward compatibility with 10.3 and 10.4 makes 64-bit Intel architecture not feasible. You are right, it is a compromise. We are making more users happy by providing a 32-bit installer for a wider range of OS releases. However, if we want a more modern certified UNIX, 64-bit installer, then we'll have to draw a line and stop supporting older OS releases. Just as we stop supporting older releases of Python. Regards, Zvezdan From ncoghlan at gmail.com Thu Apr 15 00:33:29 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Apr 2010 08:33:29 +1000 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <4BC642B9.3070106@gmail.com> Brett Cannon wrote: > And just a quick suggestion: can we standardize what > imp.source_to_path() and friend are supposed to return if the > interpreter doesn't support bytecode? I will probably have to rely on > that for something so it would be best to say now whether it should be > None or raise an exception so there is no divergence on this between VMs. Returning None sounds like the most straightforward option. "__cached__ = None" will just mean "for whatever reason, we have no cached filename for this file". It may be the cached file doesn't exist, or the interpreter simply wasn't in a position to figure it out in a user visible way. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From janssen at parc.com Thu Apr 15 01:22:36 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 14 Apr 2010 16:22:36 PDT Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19398.3934.111522.622407@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> <19398.3934.111522.622407@montanaro.dyndns.org> Message-ID: <52178.1271287356@parc.com> skip at pobox.com wrote: > Now that I think about it, it might not be sufficient to just hide those > directories from the environment. The Python setup.py file has > unconditional hard-coded references to /sw, /opt/local and /usr/local. That I added that to my virus scanner: check any setup.py files for /sw. I autopatch a couple of the commopn ones to get rid of it. Bill From barry at python.org Thu Apr 15 03:10:15 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Apr 2010 21:10:15 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <4BC642B9.3070106@gmail.com> References: <20100413162107.171c724c@heresy> <4BC642B9.3070106@gmail.com> Message-ID: <20100414211015.2df61571@heresy> On Apr 15, 2010, at 08:33 AM, Nick Coghlan wrote: >Brett Cannon wrote: >> And just a quick suggestion: can we standardize what >> imp.source_to_path() and friend are supposed to return if the >> interpreter doesn't support bytecode? I will probably have to rely on >> that for something so it would be best to say now whether it should be >> None or raise an exception so there is no divergence on this between VMs. > >Returning None sounds like the most straightforward option. "__cached__ >= None" will just mean "for whatever reason, we have no cached filename >for this file". It may be the cached file doesn't exist, or the >interpreter simply wasn't in a position to figure it out in a user >visible way. I completely agree. The PEP already leaves __cached__ up to the implementation, but I'll update it to be clear that None is an acceptable return value from imp.cached_from_source() (which is the one I think you mean), and also what __cached__=None means. Thanks Brett and Nick. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From stephen at xemacs.org Thu Apr 15 04:51:39 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 15 Apr 2010 11:51:39 +0900 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <45888.1271268083@parc.com> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> Message-ID: <87ochl2zt0.fsf@uwakimon.sk.tsukuba.ac.jp> Bill Janssen writes: > > Fink or MacPorts are often a practical necessity. > > Fink is deadly, MacPorts much more benign, in my experience. Which is > several years out-of-date, before I realized I didn't need either one of > them, and before the UNIX community started adding configure patches to > support OS X builds more widely. Perhaps they've improved. FWIW, I use MacPorts heavily, and it works fine for the most recent release of Mac OS X which has been out for at least 6 months. Bug fixes for Snow Leopard in MacPorts are typically a matter of a handful of hours, but many port maintainers don't have full-time access to Leopard, and few will do more than pay lip service and help install a patch if you have problems with Tiger. IOW, the previous release always has occasional glitches, and the second previous release is a nightmare. MacPorts is fine if you don't mind occasionally messing with it and don't need to deal with Mac OS versions getting even slightly long in the tooth. > In any case, they shouldn't be needed on buildbots maintained by > the PSF. They should be avoided, as many users won't have them. MacPorts and presumably Fink have their own testing process, and the quality of /opt/local/lib varies quite a bit. From martin at v.loewis.de Thu Apr 15 06:36:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 06:36:34 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC63457.5020006@canterbury.ac.nz> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> Message-ID: <4BC697D2.4020200@v.loewis.de> Greg Ewing wrote: > Michael Foord wrote: >> Building Python requires, I believe, the XCode development tools to be >> installed. Even then, building a full version of Python - with *all* >> the C extensions that are part of a Python release - is not a trivial >> task. > > What's non-trivial about it? Building a DMG file, in a way that the output will actually work on most other systems. Regards, Martin From glyph at twistedmatrix.com Thu Apr 15 08:24:53 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 15 Apr 2010 02:24:53 -0400 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not simply issues that had a patch ready to go. I'm curious - if one isn't supposed to ping the mailing list every time, how does one ask the tracker "please show me all the issues which have a patch ready to go that hasn't been reviewed / responded to / rejected or applied"? It seems like patches sometimes linger for quite a while and often their workflow-state is highly unclear (to me, at least). From glyph at twistedmatrix.com Thu Apr 15 08:28:13 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 15 Apr 2010 02:28:13 -0400 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not simply issues that had a patch ready to go. Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear to me that Martin specifically meant "simply issues that had a patch ready to go". Quoting him exactly: >> Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. It seems that perhaps the core developers have slightly different opinions about this? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Thu Apr 15 09:20:20 2010 From: db3l.net at gmail.com (David Bolen) Date: Thu, 15 Apr 2010 03:20:20 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> Message-ID: Ronald Oussoren writes: > Speaking of which... I have a mac-mini that could be used for a > buildbot. How much work is needed to kickstart a buildbot, taking > into account that I'd prefer to have a buildbot with different > configure-flags that the default unix build (that is, I want to test > a framework build that is a universal binary). Sort of picking this message to enter the thread ... Interestingly enough, I had made an offer to Martin to host an OSX build slave last week (before this thread), since I had a Mini whose disk crashed and would be reasonably free after I rebuilt it, and I had noticed there were no longer any OSX build slaves around. It's online now, though it certainly need not be the only one. There's not much to setting up a build slave - just easy_install buildbot on top of a basic Python install (you'll need Twisted), use the "build-slave" command to create a tree, edit the files in "info", and "buildbot start " to get it started. Of course, there's some overhead to monitoring things over time. My new slave is a Tiger box, since that's what I still need for my own application builds. So it won't help with building/testing x64 builds. Per the "startup on reboot" part of the thread, I typically use the LaunchAgents setup on my Macs for that purpose, although to be honest, not many of my build slaves start automatically anyway, as the boxes don't tend to get restarted other than manually. In the first builds, I have noticed that the build master seems to execute the builds as a Unix system, so isn't building with the universalsdk option or as a framework, though the latter would probably need a bit of glue to make the framework available just locally to the buildbot process. On Windows, the buildbot tasks use scripts that keep the third party stuff in the buildbot branch tree itself. I wonder if perhaps with a little TLC, we could come up with some Mac-specific buildbot scripts to better have an OSX build slave mimic the final build process. It might be as simple as using build-installer with a specified directory within the build tree, though utilitizing the built framework for the tests probably needs some support. The build master could then be set to execute those scripts (much as it does on Windows) rather than the stock Unix approach. I anticipate some tuning to do with respect to external libraries. Rather than use system libraries or my own (or finks or MacPorts) version of libraries, I suspect it would be better to download the same releases of the third party stuff that the installer script downloads and install them locally for normal buildbot builds, to best match what would be packaged up with a final binary. Of course, if so, then there's the question of keeping the buildbot environment up to date with the installer script. As I believe Martin mentioned elsewhere in the thread, he used to have my Windows slave generate nightly MSI builds and upload them, but the process became fragile over time, and didn't seem to be missed when we discontinued them. But something similar could probably be worked out for building nightly DMGs for OSX if it were deemed useful. > Creating the Mac installer is easy: just run > Mac/BuildScript/build-installer.py on an OSX 10.5 system where a > local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The > system should also not have fink or darwinports and a clean > /usr/local tree to avoid contaminating the build. Yes, I've successfully used the script to create a DMG of the latest 2.7 out of trunk, and at least the basic sniff test (install it locally, run a few things) seems fine. Did run into one quirk where if you have already built Python within the tree you run the script from, it doesn't rebuild everything beneath /tmp/_py/_bld - such as the pgen stuff. So you want a clean source tree too. To the extent that the output (DMG) of the build-installer script is what is currently needed for OSX, I can generate one if it is still needed, though I can't promise much beyond just executing the script. The new build slave can also be made available for RMs (or whomever) to generate the DMG when needed if that might be helpful. Though there should probably be some basic installation test on other systems prior to publishing any such generated files. -- David From techtonik at gmail.com Thu Apr 15 10:20:56 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 15 Apr 2010 11:20:56 +0300 Subject: [Python-Dev] OS information, tags In-Reply-To: <4BC46996.4020909@gmail.com> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan wrote: > >>> ? ? ? ? ? ?I am surprised to see that the bug-tracker >>> doesn't have an OS classifier or ability to add >>> tags ? Since a number of issues reported seem to Just to remind about my +1 for user editable tags. >> There is one. In the Components you can do a multiple select and it >> has Macintosh , Windows as options. > > I think that setup dates from the Sourceforge days when we didn't have > keywords or the ability to add our own fields. Would it make sense to > put a request on the metatracker to convert these to keywords now that > they're available? Keywords are not currently editable by users. > Or even a separate OS field with "Windows, Mac OS X, > Linux, *BSD, Other" as the options? It is not uncommon when code written on Unix can affect both MacOS and Windows, so you need to be able to select both. > While there is some Windows and Mac specific code, treating them as > separate components seems fairly unintuitive. -- anatoly t. From cournape at gmail.com Thu Apr 15 11:34:57 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 15 Apr 2010 18:34:57 +0900 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <19398.3934.111522.622407@montanaro.dyndns.org> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <19397.65086.457847.833679@montanaro.dyndns.org> <45888.1271268083@parc.com> <19398.3934.111522.622407@montanaro.dyndns.org> Message-ID: On Thu, Apr 15, 2010 at 3:54 AM, wrote: > > ? ?Bill> In any case, they shouldn't be needed on buildbots maintained by > ? ?Bill> the PSF. > > Sure. ?My question was related to humans building binary distributions > though. ?Unless that becomes fully automated so the release manager can just > push a button and have it built on and as-yet-nonexistent Mac OSX buildbot > machine, somebody will have to generate that installer. ?Ronald says Fink, > MacPorts and /usr/local are poison. ?If that's truly the case that's fine. > It's just that it reduces the size of the potential binary installer build > machines. Actually, you can just use a chroot "jail" to build the binary - I use this process to build the official numpy/scipy binaries, it works very well whatever crap there is on my laptop otherwise. cheers, David From nad at acm.org Thu Apr 15 13:41:03 2010 From: nad at acm.org (Ned Deily) Date: Thu, 15 Apr 2010 04:41:03 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> Message-ID: In article <4BC697D2.4020200 at v.loewis.de>, "Martin v. Lowis" wrote: > Greg Ewing wrote: > > Michael Foord wrote: > >> Building Python requires, I believe, the XCode development tools to be > >> installed. Even then, building a full version of Python - with *all* > >> the C extensions that are part of a Python release - is not a trivial > >> task. > > What's non-trivial about it? > Building a DMG file, in a way that the output will actually work on most > other systems. As Ronald pointed out, the installer build script does all of the dirty work of building the install disk image (the .dmg file), including downloading and building necessary third-party libraries. What isn't automatically checked at the moment is that the third-party Tk framework is in place during the build and that there isn't contamination from the build user's environment. There is a patch in Issue5651 to do much of that. It doesn't currently try to handle the possibility of MacPorts (/opt/local) and Fink (/sw) contamination. I believe that issue should be addressed by the resolution of Issue7713. Keep in mind that Python on OS X supports the standard OS X multi-architecture (Intel vs PPC and/or 32-bit vs 64-bit executables in the same file) and multi-version features (the current installer builds are fully supported on 10.6, 10.5, and 10.4 and "best-effort" supported on 10.3 as well although building is not supported on 10.3) as well as "Unix" shared library vs OS X framework shared library configurations. That leads to a rather imposing matrix of potential build systems vs potential target systems. For the most part, Apple provides excellent upward compatibility; downward is somewhat trickier. OS X 10.6 (Snow Leopard) has complicated matters by the change to preferring 64-bit building and executing where possible. For python builds, some build configurations that worked by default under 10.5 and earlier no longer do so on 10.6, primarily due to standard library modules that depend on APIs, in many cases deprecated ones, that are only available in 32-bit versions. The old Macintosh modules comprise the biggest group of these and, while they have been removed in 3.x, they still remain in 2.x. But there are some others as well, which explain most of the build issues Michael reported. There are also newer versions of some open source libraries in 10.6 which cause problems for multi-version builds: openssl, for one, and Apple's 64-bit version of Tk 8.5. None of these problems are insolvable but with the very limited resources (i.e. people time) we've had for working on these issues, and when there have been so many other more critical problems, it has been easier up to now to stick with building on 10.5 (or even on 10.4 which I test build occasionally). I think the single most important thing that can be done to help right now is to get a robust system of OS X buildbots going so that many of the critical problems can be detected up front rather than when one of us gets around to building and install testing the multi-arch and multi-version installers. Since there are so many potential configurations, some thought needs to go into which would be of the most value. I would say that the most important build configurations are: (1) 32-bit Intel/PPC framework on 10.5 (and eventually 10.6) targeted for 10.3 through 10.6 (the current installer config setup); and (2) 32-/64- Intel-only framework for 10.6; followed by (3) 32-/64- Intel/PPC framework for 10.5 and 10.6. Building the whole installer and testing on as many targeted systems as possible would be ideal but that adds more complexity to the automation (again, not insurmountable). But even just doing framework multi-arch builds, installs, and tests (using the appropriate options) on only the build systems themselves without building an installer or third-party libs would go a long way towards catching many, if not most, problems early. I'd be happy to supply more detailed suggestions for specific configuration parameters for those interested in setting up buildbots. (There may be some delay, though, as I will have limited time and Internet access for the next three weeks.) -- Ned Deily, nad at acm.org From nad at acm.org Thu Apr 15 14:18:59 2010 From: nad at acm.org (Ned Deily) Date: Thu, 15 Apr 2010 05:18:59 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> Message-ID: In article <4BC63599.5020005 at voidspace.org.uk>, Michael Foord wrote: > A build on my machine produces output similar to: > > > Python build finished, but the necessary bits to build these modules > were not found: > _bsddb third-party (Sleepycat) library needed (see the installer script) > dl only available for 32-bit > gdbm third-party library needed (not supplied in the installer build) > imageop only available for 32-bit (and removed in 3.x) > linuxaudiodev ossaudiodev neither supported on OS X > readline requires either GNU readline lib (not included with OS X) or (with 2.6.5, trunk, or py3k) can now use OS X editline (libedit) replacement when targeting for 10.5 or 10.6 (add MACOSX_DEPLOYMENT_TARGET=10.5 or 10.6 to ./configure arguments). > spwd sunaudiodev neither supported on OS X > Failed to build these modules: > _tkinter currently only supported for 32-bit archs as there was no 64-bit (non-X) Tk on OS X prior to 10.6 and there are unresolved problems with the 10.6 Apple-supplied Tk 8.5 and 2.6.x IDLE (Issue6864). -- Ned Deily, nad at acm.org From nad at acm.org Thu Apr 15 14:21:16 2010 From: nad at acm.org (Ned Deily) Date: Thu, 15 Apr 2010 05:21:16 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> Message-ID: In article <4BC61278.7020108 at v.loewis.de>, "Martin v. Lowis" wrote: > Ned Deily wrote: > > That *is* something that the PSF could help with. I > > would be happy to help with that myself, although my time to do so will > > be very limited for the next few weeks. > > The PSF still has a machine that was donated by Apple that once used to > be a build slave. Unfortunately, that machine had hardware problems (or > atleast appeared to have hardware problems). So if anybody would be > interested into maintaining it, please let us know. Any idea what type of machine it is and where it is currently located? -- Ned Deily, nad at acm.org From fuzzyman at voidspace.org.uk Thu Apr 15 15:01:19 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 15 Apr 2010 15:01:19 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> Message-ID: <4BC70E1F.3060304@voidspace.org.uk> Whilst making Python easier to build on the Mac is certainly a worthy goal, the point of my post was to demonstrate (in reply to an email by Greg Ewing) *why* building a *full* Python from source was non-trivial. I personally only build Python from source to test changes to core-Python and am happy with the Python binary installers for the Mac - once they arrive. :-) All the best, Michael On 15/04/2010 14:18, Ned Deily wrote: > In article<4BC63599.5020005 at voidspace.org.uk>, > Michael Foord wrote: > >> A build on my machine produces output similar to: >> >> >> Python build finished, but the necessary bits to build these modules >> were not found: >> _bsddb >> > third-party (Sleepycat) library needed (see the installer script) > > >> dl >> > only available for 32-bit > > >> gdbm >> > third-party library needed (not supplied in the installer build) > > >> imageop >> > only available for 32-bit (and removed in 3.x) > > >> linuxaudiodev ossaudiodev >> > neither supported on OS X > > >> readline >> > requires either GNU readline lib (not included with OS X) or (with > 2.6.5, trunk, or py3k) can now use OS X editline (libedit) replacement > when targeting for 10.5 or 10.6 (add MACOSX_DEPLOYMENT_TARGET=10.5 or > 10.6 to ./configure arguments). > > >> spwd sunaudiodev >> > neither supported on OS X > > >> Failed to build these modules: >> _tkinter >> > currently only supported for 32-bit archs as there was no 64-bit (non-X) > Tk on OS X prior to 10.6 and there are unresolved problems with the 10.6 > Apple-supplied Tk 8.5 and 2.6.x IDLE (Issue6864). > > -- http://www.ironpythoninaction.com/ From brian.curtin at gmail.com Thu Apr 15 16:11:51 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 15 Apr 2010 09:11:51 -0500 Subject: [Python-Dev] Status of 2.7b1? In-Reply-To: References: <4BC09E77.8030209@gmail.com> <4BC0C55A.5060001@gmail.com> Message-ID: On Tue, Apr 13, 2010 at 14:43, David Bolen wrote: > Brian Curtin writes: > > > The tests are run on a native Win32 build as compiled by VS2008. The > > functionality is Win32 specific and wouldn't work on Cygwin, so the tests > > are skipped there. I believe Cygwin is used for kicking off the tests and > > other buildbot stuff, but they don't actually run through Cygwin. > > Yes, that's correct. Cygwin is on my buildbot just for management > convenience. The build slave happens to be started from a Cygwin bash > shell (and beneath a Cygwin-based home directory, which is why you see > it in various path references in logs), but the buildbot code itself > is responsible for executing the python process directly for testing. > And python itself is built normally with VS 2008 as a pure win32 > build, no Cygwin dependency. > > I did also verify over the weekend that the failures occur whether the > python_d process is started from a Cygwin bash shell or from the > normal Windows cmd process, and even if test_os is used directly, > without running through rt.bat. The case that ran successfully was > not via test_os, but only by interactively replicating the test. > > I'm back at this point, and to the extent my buildbot is the only > place currently replicating the problem, am working with Brian for > whatever access or resources might be helpful. > > -- David Although the fix still needs a bit of cleanup, test_os seems to be working fine on this buildbot now. The test now waits for the subprocess to communicate back that it is running, rather than the time.sleep hacks, then it goes forward with os.kill. Big thanks to David for access to the machine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Apr 15 16:13:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Apr 2010 00:13:12 +1000 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: <4BC71EF8.1030305@gmail.com> anatoly techtonik wrote: > On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan wrote: >> Or even a separate OS field with "Windows, Mac OS X, >> Linux, *BSD, Other" as the options? > > It is not uncommon when code written on Unix can affect both MacOS and > Windows, so you need to be able to select both. Yep, if adopted, it would be a multi-select field. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Apr 15 16:31:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Apr 2010 14:31:23 +0000 (UTC) Subject: [Python-Dev] OS information, tags References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <4BC71EF8.1030305@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > anatoly techtonik wrote: > > On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan gmail.com>wrote: > >> Or even a separate OS field with "Windows, Mac OS X, > >> Linux, *BSD, Other" as the options? > > > > It is not uncommon when code written on Unix can affect both MacOS and > > Windows, so you need to be able to select both. > > Yep, if adopted, it would be a multi-select field. We already have "Macintosh" and "Windows" in the multi-select component field. It would be nice if the bug interface didn't grow more complicated than it already is. Thanks Antoine. From ncoghlan at gmail.com Thu Apr 15 16:45:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Apr 2010 00:45:34 +1000 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <4BC71EF8.1030305@gmail.com> Message-ID: <4BC7268E.5080003@gmail.com> Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> anatoly techtonik wrote: >>> On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan gmail.com>wrote: >>>> Or even a separate OS field with "Windows, Mac OS X, >>>> Linux, *BSD, Other" as the options? >>> It is not uncommon when code written on Unix can affect both MacOS and >>> Windows, so you need to be able to select both. >> Yep, if adopted, it would be a multi-select field. > > We already have "Macintosh" and "Windows" in the multi-select component field. > It would be nice if the bug interface didn't grow more complicated than it > already is. That's where the discussion started - whether or not moving that information out to a separate field would actually be an improvement or not. I don't use the tracker intensively enough to give a considered opinion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Apr 15 16:53:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Apr 2010 14:53:35 +0000 (UTC) Subject: [Python-Dev] OS information, tags References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <4BC71EF8.1030305@gmail.com> <4BC7268E.5080003@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > That's where the discussion started - whether or not moving that > information out to a separate field would actually be an improvement or > not. I don't use the tracker intensively enough to give a considered > opinion. For me, the less UI overhead the better. If people want to search those issues and the Roundup search interface isn't practical enough (which I understand - most issue trackers seem to have awful search interfaces), we can predefine custom searches for Windows and Mac issues, and publish them in the left bar. Regards Antoine. From orsenthil at gmail.com Thu Apr 15 17:00:47 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 15 Apr 2010 20:30:47 +0530 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <4BC71EF8.1030305@gmail.com> Message-ID: <20100415150047.GA9617@remy> On Thu, Apr 15, 2010 at 02:31:23PM +0000, Antoine Pitrou wrote: > We already have "Macintosh" and "Windows" in the multi-select component field. > It would be nice if the bug interface didn't grow more complicated than it > already is. +1 There isn't any need for yet another classification. -- Senthil Your life would be very empty if you had nothing to regret. From brian.curtin at gmail.com Thu Apr 15 17:09:03 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 15 Apr 2010 10:09:03 -0500 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: On Thu, Apr 15, 2010 at 03:20, anatoly techtonik wrote: > On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan wrote: > > > >>> I am surprised to see that the bug-tracker > >>> doesn't have an OS classifier or ability to add > >>> tags ? Since a number of issues reported seem to > > Just to remind about my +1 for user editable tags. > -sys.maxint on just about anything user editable except for the title or messages on an issue. I don't think user editable tags on an issue bring anything to the table except making it seem more "web 2.0". Searching user-generated tags would be a nightmare because you'd have to know every combination of some idea in order to get relevant results (e.g., windows, windoze, window$, win32). -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Thu Apr 15 17:09:47 2010 From: skip at pobox.com (skip at pobox.com) Date: Thu, 15 Apr 2010 10:09:47 -0500 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> Message-ID: <19399.11323.946604.992452@montanaro.dyndns.org> Ned> Any idea what type of machine it is and where it is currently Ned> located? I seem to recall it is/was a G4 XServe. My guess as to location would be at xs4all.nl. Skip From orsenthil at gmail.com Thu Apr 15 18:07:18 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 15 Apr 2010 21:37:18 +0530 Subject: [Python-Dev] urlparse - to parse IPv6 URL in 2.7 b2 ? Message-ID: <20100415160718.GA12597@remy> http://bugs.python.org/issue2987 This deals with a feature request of parsing an IPv6 URL according to standards. The patch is pretty complete and we have good test coverage too. Is it okay to include this in Python 2.7 b2 release? It would be a worthy addition. -- Senthil From brett at python.org Thu Apr 15 20:55:10 2010 From: brett at python.org (Brett Cannon) Date: Thu, 15 Apr 2010 11:55:10 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: Yes, we have different opinions. My personal take is to wait a week before you email python-dev if there has been no activity. That is enough time for people interested in the patch to get to it as we all have different schedules. Any faster and it feels like noise on the list to me. Brett (from his phone) On Apr 14, 2010 11:28 PM, "Glyph Lefkowitz" wrote: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not sim... Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear to me that Martin specifically meant "simply issues that had a patch ready to go". Quoting him exactly: Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. It seems that perhaps the core developers have slightly different opinions about this? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Apr 15 22:13:49 2010 From: brett at python.org (Brett Cannon) Date: Thu, 15 Apr 2010 13:13:49 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: Yes, we have different opinions. My personal take is to wait a week before you email python-dev if there has been no activity. That is enough time for people interested in the patch to get to it as we all have different schedules. Any faster and it feels like noise on the list to me. Brett (from his phone) On Apr 14, 2010 11:28 PM, "Glyph Lefkowitz" wrote: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not sim... Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear to me that Martin specifically meant "simply issues that had a patch ready to go". Quoting him exactly: Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. It seems that perhaps the core developers have slightly different opinions about this? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Apr 15 22:39:15 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 22:39:15 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> Message-ID: <4BC77973.1080803@v.loewis.de> >>> What's non-trivial about it? >> Building a DMG file, in a way that the output will actually work on most >> other systems. > > As Ronald pointed out, the installer build script does all of the dirty > work of building the install disk image (the .dmg file), including > downloading and building necessary third-party libraries. Hmm. When I tried it last, it didn't do anything - it just crashed. I then had to fix it, and also arrange to meet a certain number of assumptions that it assumed but that had not been setup on my system. This is some years ago, so I don't recall details. > I think the single most important thing that > can be done to help right now is to get a robust system of OS X > buildbots going so that many of the critical problems can be detected up > front rather than when one of us gets around to building and install > testing the multi-arch and multi-version installers. There is one build slave now up, contributed by David Bolen. Regards, Martin From martin at v.loewis.de Thu Apr 15 22:40:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 22:40:39 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> Message-ID: <4BC779C7.7080001@v.loewis.de> Ned Deily wrote: > In article <4BC61278.7020108 at v.loewis.de>, > "Martin v. Lowis" wrote: >> Ned Deily wrote: >>> That *is* something that the PSF could help with. I >>> would be happy to help with that myself, although my time to do so will >>> be very limited for the next few weeks. >> The PSF still has a machine that was donated by Apple that once used to >> be a build slave. Unfortunately, that machine had hardware problems (or >> atleast appeared to have hardware problems). So if anybody would be >> interested into maintaining it, please let us know. > > Any idea what type of machine it is and where it is currently located? No idea about the former, except that it is an XServe computer. It is located in Amsterdam (in the Netherlands), in the XS4ALL facility. Regards, Martin From martin at v.loewis.de Thu Apr 15 22:48:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 22:48:49 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> Message-ID: <4BC77BB1.3070106@v.loewis.de> > In the first builds, I have noticed that the build master seems to > execute the builds as a Unix system, so isn't building with the > universalsdk option or as a framework, though the latter would > probably need a bit of glue to make the framework available just > locally to the buildbot process. Not sure what you mean by "make available" - I thought this is just a matter of configure options? Would you like to see such a build on your machine in addition, or instead of, the Unix build? > On Windows, the buildbot tasks use scripts that keep the third party > stuff in the buildbot branch tree itself. I wonder if perhaps with a > little TLC, we could come up with some Mac-specific buildbot scripts > to better have an OSX build slave mimic the final build process. No. I'd rather create a separate builder on the master. > I anticipate some tuning to do with respect to external libraries. > Rather than use system libraries or my own (or finks or MacPorts) > version of libraries, I suspect it would be better to download the > same releases of the third party stuff that the installer script > downloads and install them locally for normal buildbot builds, to best > match what would be packaged up with a final binary. Of course, if > so, then there's the question of keeping the buildbot environment up > to date with the installer script. It would be possible to commit these packages into the externals repository, just as we do for Windows. It would then be possible to check them out over again all the time, or somehow to detect when the URL changes so they export a different subversion directory. > To the extent that the output (DMG) of the build-installer script is > what is currently needed for OSX, I can generate one if it is still > needed, though I can't promise much beyond just executing the script. > The new build slave can also be made available for RMs (or whomever) > to generate the DMG when needed if that might be helpful. Though > there should probably be some basic installation test on other systems > prior to publishing any such generated files. For 2.7, I would do that only if the very same build process is also going to be used for the final release. There is no point in testing builds when then the final release uses some other process. It would also be good if anybody who commits to producing OSX binaries now would also produce them throughout the 2.7 lifetime (which could be a bit longer than the traditional 18 months). Regards, Martin From martin at v.loewis.de Thu Apr 15 22:53:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 22:53:37 +0200 Subject: [Python-Dev] urlparse - to parse IPv6 URL in 2.7 b2 ? In-Reply-To: <20100415160718.GA12597@remy> References: <20100415160718.GA12597@remy> Message-ID: <4BC77CD1.1060203@v.loewis.de> Senthil Kumaran wrote: > http://bugs.python.org/issue2987 > > This deals with a feature request of parsing an IPv6 URL according to > standards. The patch is pretty complete and we have good test coverage > too. > > Is it okay to include this in Python 2.7 b2 release? It would be a > worthy addition. 1. The ultimate decision on that is with the release manager (i.e. Benjamin) 2. My personal recommendation is to allow it: while it *is* a new feature, it is a feature that doesn't change any API. Regards, Martin From solipsis at pitrou.net Thu Apr 15 23:07:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Apr 2010 21:07:23 +0000 (UTC) Subject: [Python-Dev] urlparse - to parse IPv6 URL in 2.7 b2 ? References: <20100415160718.GA12597@remy> <4BC77CD1.1060203@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > [...] > 2. My personal recommendation is to allow it: while it *is* a new > feature, it is a feature that doesn't change any API. That's my opinion too. Regards Antoine. From nad at acm.org Thu Apr 15 23:12:07 2010 From: nad at acm.org (Ned Deily) Date: Thu, 15 Apr 2010 14:12:07 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> Message-ID: In article <19399.11323.946604.992452 at montanaro.dyndns.org>, skip at pobox.com wrote: > Ned> Any idea what type of machine it is and where it is currently > Ned> located? > > I seem to recall it is/was a G4 XServe. My guess as to location would be at > xs4all.nl. If it were working that could be of use. It would not be able to run OS X 10.6 but having a 10.5 system PPC system as a buildbot would certainly be useful; it should be fine for the default installer configuration builds. (Alas, I don't expect to be anywhere in the vicinity in the foreseeable future.) -- Ned Deily, nad at acm.org From martin at v.loewis.de Thu Apr 15 23:35:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Apr 2010 23:35:30 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> Message-ID: <4BC786A2.8040109@v.loewis.de> > If it were working that could be of use. It would not be able to run OS > X 10.6 but having a 10.5 system PPC system as a buildbot would certainly > be useful; it should be fine for the default installer configuration > builds. (Alas, I don't expect to be anywhere in the vicinity in the > foreseeable future.) I think we could arrange to have somebody ship it somewhere, even across the globe. Regards, Martin From brett at python.org Thu Apr 15 23:51:44 2010 From: brett at python.org (Brett Cannon) Date: Thu, 15 Apr 2010 14:51:44 -0700 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> Message-ID: On Thu, Apr 15, 2010 at 04:41, Ned Deily wrote: > In article <4BC697D2.4020200 at v.loewis.de>, > "Martin v. Lowis" wrote: > > Greg Ewing wrote: > > > Michael Foord wrote: > > >> Building Python requires, I believe, the XCode development tools to be > > >> installed. Even then, building a full version of Python - with *all* > > >> the C extensions that are part of a Python release - is not a trivial > > >> task. > > > What's non-trivial about it? > > Building a DMG file, in a way that the output will actually work on most > > other systems. > > As Ronald pointed out, the installer build script does all of the dirty > work of building the install disk image (the .dmg file), including > downloading and building necessary third-party libraries. What isn't > automatically checked at the moment is that the third-party Tk framework > is in place during the build and that there isn't contamination from the > build user's environment. There is a patch in Issue5651 to do much of > that. It doesn't currently try to handle the possibility of MacPorts > (/opt/local) and Fink (/sw) contamination. I believe that issue should > be addressed by the resolution of Issue7713. > I know for me the reason I have never tried to help with building the binaries is I simply lack the expertise. I always build straight UNIX versions of Python under OS X, so I have no experience with framework builds (which is what the binary distribution is). Heck, I didn't even know about the build script until just now since it's such a pain to build. Plus I don't know if you can use a framework build from within your svn checkout for testing, so I REALLY don't bother building a framework. And to top it off I use the latest libraries of things to look for possible breakage. I am sure I am not the only person who has all of these barriers preventing them from using a framework build consistently. > > Keep in mind that Python on OS X supports the standard OS X > multi-architecture (Intel vs PPC and/or 32-bit vs 64-bit executables in > the same file) and multi-version features (the current installer builds > are fully supported on 10.6, 10.5, and 10.4 and "best-effort" supported > on 10.3 as well although building is not supported on 10.3) as well as > "Unix" shared library vs OS X framework shared library configurations. > That leads to a rather imposing matrix of potential build systems vs > potential target systems. For the most part, Apple provides excellent > upward compatibility; downward is somewhat trickier. OS X 10.6 (Snow > Leopard) has complicated matters by the change to preferring 64-bit > building and executing where possible. For python builds, some build > configurations that worked by default under 10.5 and earlier no longer > do so on 10.6, primarily due to standard library modules that depend on > APIs, in many cases deprecated ones, that are only available in 32-bit > versions. The old Macintosh modules comprise the biggest group of these > and, while they have been removed in 3.x, they still remain in 2.x. But > there are some others as well, which explain most of the build issues > Michael reported. There are also newer versions of some open source > libraries in 10.6 which cause problems for multi-version builds: > openssl, for one, and Apple's 64-bit version of Tk 8.5. > > None of these problems are insolvable but with the very limited > resources (i.e. people time) we've had for working on these issues, and > when there have been so many other more critical problems, it has been > easier up to now to stick with building on 10.5 (or even on 10.4 which I > test build occasionally). I think the single most important thing that > can be done to help right now is to get a robust system of OS X > buildbots going so that many of the critical problems can be detected up > front rather than when one of us gets around to building and install > testing the multi-arch and multi-version installers. Since there are so > many potential configurations, some thought needs to go into which would > be of the most value. I would say that the most important build > configurations are: (1) 32-bit Intel/PPC framework on 10.5 (and > eventually 10.6) targeted for 10.3 through 10.6 (the current installer > config setup); and (2) 32-/64- Intel-only framework for 10.6; followed > by (3) 32-/64- Intel/PPC framework for 10.5 and 10.6. Building the > whole installer and testing on as many targeted systems as possible > would be ideal but that adds more complexity to the automation (again, > not insurmountable). But even just doing framework multi-arch builds, > installs, and tests (using the appropriate options) on only the build > systems themselves without building an installer or third-party libs > would go a long way towards catching many, if not most, problems early. > I'd be happy to supply more detailed suggestions for specific > configuration parameters for those interested in setting up buildbots. > (There may be some delay, though, as I will have limited time and > Internet access for the next three weeks.) > > And this is another reason why Ronald has always been the only person to build the OS X binary; it's complicated. Knowing how to get the proper universal framework with the right things is annoying. Plus making sure something works on multiple versions of OS X is a pain as I am sure very few of us have 10.6, 10.5, and 10.4 lying around. I mean if the build script was dead-simple run and forget I am sure many of us would be willing to create dmgs. But it will take someone who knows what is needed (sounds like Ned does and obviously Ronald does) to get it to that place. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Apr 16 00:05:13 2010 From: brett at python.org (Brett Cannon) Date: Thu, 15 Apr 2010 15:05:13 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: Yes, we have different opinions. My personal take is to wait a week before you email python-dev if there has been no activity. That is enough time for people interested in the patch to get to it as we all have different schedules. Any faster and it feels like noise on the list to me. Brett (from his phone) On Apr 14, 2010 11:28 PM, "Glyph Lefkowitz" wrote: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not sim... Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear to me that Martin specifically meant "simply issues that had a patch ready to go". Quoting him exactly: Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. It seems that perhaps the core developers have slightly different opinions about this? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Fri Apr 16 00:37:42 2010 From: db3l.net at gmail.com (David Bolen) Date: Thu, 15 Apr 2010 18:37:42 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > Not sure what you mean by "make available" - I thought this is just a > matter of configure options? Building as a framework, yes. But I think there's some steps to take to then have the test python binary use the locally built framework while running the tests (since the framework won't be in a system location like /Library/Frameworks during the test). Probably similar to whatever py2app does to use packaged frameworks contained within the application bundle. I doubt it's overly complex, but just an extra step that might need to be incorporated into any buildbot test rules or script to ensure the just built framework is used in the tests. > Would you like to see such a build on your machine in addition, or > instead of, the Unix build? I suppose an argument could be made to test both, since others have indicated here they often do a normal Unix-oriented build on OSX, but I think if I only had to pick one, I'd go with framework to match what gets published as a binary for downloading. Theoretically it should probably also be a universal build, though of course only one variant would actually get tested on a build slave depending on its platform. Certainly anything we could do to make the buildbot generated builds match as closely to possible the distribution binary process would be beneficial, I'd think. >> On Windows, the buildbot tasks use scripts that keep the third party >> stuff in the buildbot branch tree itself. I wonder if perhaps with a >> little TLC, we could come up with some Mac-specific buildbot scripts >> to better have an OSX build slave mimic the final build process. > > No. I'd rather create a separate builder on the master. I'm wondering if I'm saying the same thing but just not using the right vernacular. Something on the master (a builder?) for Windows instructs it to run Tools\buildbot\build.bat for building and test.bat for testing, rather than each of the individual commands. Whether we encapsulate the needed logic in separate makefile targets for the Mac, or independent build scripts like on Windows, the build master would only need to execute some Mac appropriate command on a given branch (not sure if that's separate builders), with the makefile/script in the given checkout handling the third party stuff and environmental setup. Keeping the knowledge in the makefile or script in the source tree would let the rules change across branches without affecting the build master. Though if having more specific rules in the master was easier, I'd be fine with that too. > It would be possible to commit these packages into the externals > repository, just as we do for Windows. It would then be possible to > check them out over again all the time, or somehow to detect when the > URL changes so they export a different subversion directory. Sure - that's the sort of thing I figured a build script could take care of, much as it does on Windows. Or, the current Mac build-installer script already has the necessary information, and can be instructed where to place the third party stuff, so it might only be necessary to have the buildbot process run that script with appropriate build tree-relative paths. > For 2.7, I would do that only if the very same build process is also > going to be used for the final release. There is no point in testing > builds when then the final release uses some other process. Agreed, thus my caveat as to the output of the build-installer script in fact being what has been published for downloads. In my brief tests it looks like it created a production DMG ready for publishing on the download page, and did in fact install correctly, but I'm still new to the Mac binary building process. > It would > also be good if anybody who commits to producing OSX binaries now would > also produce them throughout the 2.7 lifetime (which could be a bit > longer than the traditional 18 months). To the extent that the installer script is in fact the definitive process, and if the machine (10.4 Intel) is considered suitable as a binary build platform, then worst case we could probably have the buildbot run the script and upload the result when needed (sort of a one-shot version of the old MSI daily builder). I don't plan on the machine going away any time soon. Of course, this is just the DMG construction. Not sure what amount of "test the installer" testing should be required prior to publication. -- David From guido at python.org Fri Apr 16 05:01:38 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Apr 2010 20:01:38 -0700 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100413162107.171c724c@heresy> References: <20100413162107.171c724c@heresy> Message-ID: Comments inline. Nothing showstopping, mostly just spewing obscure background information... Overall, congratulations! I'm fine with the implementation going in and the PEP being marked as accepted as long as you get to the clarifications I suggest below soon after. --Guido On Tue, Apr 13, 2010 at 1:21 PM, Barry Warsaw wrote: > I am attaching the latest revision of PEP 3147 to this message, which is also > available here: > > http://www.python.org/dev/peps/pep-3147/ > > I think the PEP is ready for pronouncement, and the patch is pretty much ready > for merging into py3k. ?The only thing that I can think of that is not > implemented yet is this section on PEP 302 loaders: > > ? ?PEP 302 [18]_ defined loaders have a `.get_filename()` method which > ? ?points to the `__file__` for a module. ?As part of this PEP, we will > ? ?extend this API, to include a new method `.get_paths()` which will > ? ?return a 2-tuple containing the path to the source file and the path > ? ?to where the matching `pyc` file is (or would be). > > I'm honestly not sure whether this is still essential, or whether the > importlib ABC changes Brett and I talked about at Pycon are still required. ?I > now believe they are at best a minor part of the implementation if so. ?Maybe > Brett can chime in on that. Fine with me to omit. > Everything else is implemented, tested, and has undergone four rounds of > Rietveld reviews (thanks Antoine, Benjamin, Brett, and Georg!). ?A fifth patch > set has been uploaded and is available here: > > http://codereview.appspot.com/842043/show TL;DR :-) > This addresses all previous comments, includes some fixes from Brian Curtin > for Windows (thanks!) and fixes __main__ and -m support. ?I'd like to commit > this to py3k sooner rather than later so that we can shake out any additional > issues that might crop up, without having to continue to maintain my external > branches. > > Guido, what say you? > -Barry > > PEP: 3147 > Title: PYC Repository Directories > Version: $Revision: 80025 $ > Last-Modified: $Date: 2010-04-12 22:17:40 -0400 (Mon, 12 Apr 2010) $ > Author: Barry Warsaw > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2009-12-16 > Python-Version: 3.2 > Post-History: 2010-01-30, 2010-02-25, 2010-03-03, 2010-04-12 > > > Abstract > ======== > > This PEP describes an extension to Python's import mechanism which > improves sharing of Python source code files among multiple installed > different versions of the Python interpreter. ?It does this by > allowing more than one byte compilation file (.pyc files) to be > co-located with the Python source file (.py file). ?The extension > described here can also be used to support different Python > compilation caches, such as JIT output that may be produced by an > Unladen Swallow [1]_ enabled C Python. > > > Background > ========== > > CPython compiles its source code into "byte code", and for performance > reasons, it caches this byte code on the file system whenever the > source file has changes. ?This makes loading of Python modules much > faster because the compilation phase can be bypassed. ?When your > source file is `foo.py`, CPython caches the byte code in a `foo.pyc` > file right next to the source. > > Byte code files contain two 32-bit numbers followed by the marshaled big-endian > [2]_ code object. ?The 32-bit numbers represent a magic number and a > timestamp. ?The magic number changes whenever Python changes the byte > code format, e.g. by adding new byte codes to its virtual machine. > This ensures that pyc files built for previous versions of the VM > won't cause problems. ?The timestamp is used to make sure that the pyc > file is not older than the py file that was used to create it. ?When is not older than -> matches (Obscure fact: the timestamp in the pyc file must match the source's mtime exactly.) > either the magic number or timestamp do not match, the py file is > recompiled and a new pyc file is written. > > In practice, it is well known that pyc files are not compatible across > Python major releases. ?A reading of import.c [3]_ in the Python > source code proves that within recent memory, every new CPython major > release has bumped the pyc magic number. > > > Rationale > ========= > > Linux distributions such as Ubuntu [4]_ and Debian [5]_ provide more > than one Python version at the same time to their users. ?For example, > Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, > with Python 2.6 being the default. > > This causes a conflict for Python source files installed by the > system (including third party packages), because you cannot compile a I'd say only 3rd part packages right? (And code written by the distro, which from Python's POV is also 3rd party.) At least ought to clarify that the stdlib is unaffected by this conflict, because multiple versions of the stdlib *are* installed. > single Python source file for more than one Python version at a time. > Thus if your system wanted to install a `/usr/share/python/foo.py`, it > could not create a `/usr/share/python/foo.pyc` file usable across all > installed Python versions. Note that (due to the magic#) Python doesn't crash, it just falls back on the slower approach of compiling from source. Perhaps more important is that different Python versions (if the user has write permission) will fight over the pyc file and rewrite it each time the source is compiled. Worse, even though the magic# is initially written as zero and then rewritten with the correct value, concurrent processes running different Python versions can actually end up reading corrupt bytecode. (Alex Martelli diagnosed this at Google years ago.) > Furthermore, in order to ease the burden on operating system packagers > for these distributions, the distribution packages do not contain > Python version numbers [6]_; they are shared across all Python > versions installed on the system. ?Putting Python version numbers in > the packages would be a maintenance nightmare, since all the packages > - *and their dependencies* - would have to be updated every time a new > Python release was added or removed from the distribution. ?Because of > the sheer number of packages available, this amount of work is > infeasible. > > C extensions can be source compatible across multiple versions of > Python. ?Compiled extension modules are usually not compatible though, Actually we typically make every effort to support backwards compatibility for compiled modules, and the module initialization API contains a version# check. This is a different version# than the import magic# and historically has changed much less frequently. > and PEP 384 [7]_ has been proposed to address this by defining a > stable ABI for extension modules. > > Because these distributions cannot share pyc files, elaborate > mechanisms have been developed to put the resulting pyc files in > non-shared locations while the source code is still shared. ?Examples > include the symlink-based Debian regimes python-support [8]_ and > python-central [9]_. ?These approaches make for much more complicated, > fragile, inscrutable, and fragmented policies for delivering Python > applications to a wide range of users. ?Arguably more users get Python > from their operating system vendor than from upstream tarballs. ?Thus, > solving this pyc sharing problem for CPython is a high priority for > such vendors. > > This PEP proposes a solution to this problem. > > > Proposal > ======== > > Python's import machinery is extended to write and search for byte > code cache files in a single directory inside every Python package > directory. ?This directory will be called `__pycache__`. > Further, pyc files will contain a magic string that differentiates the Clarify that the magic string is in the filename, not in the file contents. > Python version they were compiled for. ?This allows multiple byte > compiled cache files to co-exist for a single Python source file. > > This scheme has the added benefit of reducing the clutter in a Python > package directory. > > When a Python source file is imported for the first time, a > `__pycache__` directory will be created in the package directory, if Is this still true? ISTR there was a lot of discussion about the auto-creation and possible security concerns. > one does not already exist. ?The pyc file for the imported source will > be written to the `__pycache__` directory, using the magic-tag By now the magic-tag format should have been defined (or a "see below" inserted). > formatted name. ?If either the creation of the `__pycache__` directory > or the pyc file inside that fails, the import will still succeed, just > as it does in a pre-PEP-3147 world. > > If the py source file is missing, the pyc file inside `__pycache__` > will be ignored. ?This eliminates the problem of accidental stale pyc > file imports. > > For backward compatibility, Python will still support pyc-only > distributions, however it will only do so when the pyc file lives in > the directory where the py file *would* have been, i.e. not in the > `__pycache__` directory. ?pyc file outside of `__pycache__` will only > be imported if the py source file is missing. > > Tools such as `py_compile` [15]_ and `compileall` [16]_ will be > extended to create PEP 3147 formatted layouts automatically, but will > have an option to create pyc-only distribution layouts. > > > Examples > --------- > > What would this look like in practice? > > Let's say we have a Python package named `alpha` which contains a > sub-package name `beta`. ?The source directory layout before byte > compilation might look like this:: > > ? ?alpha/ > ? ? ? ?__init__.py > ? ? ? ?one.py > ? ? ? ?two.py > ? ? ? ?beta/ > ? ? ? ? ? ?__init__.py > ? ? ? ? ? ?three.py > ? ? ? ? ? ?four.py > > After byte compiling this package with Python 3.2, you would see the > following layout:: > > ? ?alpha/ > ? ? ? ?__pycache__/ > ? ? ? ? ? ?__init__.cpython-32.pyc > ? ? ? ? ? ?one.cpython-32.pyc > ? ? ? ? ? ?two.cpython-32.pyc > ? ? ? ?__init__.py > ? ? ? ?one.py > ? ? ? ?two.py > ? ? ? ?beta/ > ? ? ? ? ? ?__pycache__/ > ? ? ? ? ? ? ? ?__init__.cpython-32.pyc > ? ? ? ? ? ? ? ?three.cpython-32.pyc > ? ? ? ? ? ? ? ?four.cpython-32.pyc > ? ? ? ? ? ?__init__.py > ? ? ? ? ? ?three.py > ? ? ? ? ? ?four.py > > *Note: listing order may differ depending on the platform.* > > Let's say that two new versions of Python are installed, one is Python > 3.3 and another is Unladen Swallow. ?After byte compilation, the file > system would look like this:: > > ? ?alpha/ > ? ? ? ?__pycache__/ > ? ? ? ? ? ?__init__.cpython-32.pyc > ? ? ? ? ? ?__init__.cpython-33.pyc > ? ? ? ? ? ?__init__.unladen-10.pyc > ? ? ? ? ? ?one.cpython-32.pyc > ? ? ? ? ? ?one.cpython-33.pyc > ? ? ? ? ? ?one.unladen-10.pyc > ? ? ? ? ? ?two.cpython-32.pyc > ? ? ? ? ? ?two.cpython-33.pyc > ? ? ? ? ? ?two.unladen-10.pyc > ? ? ? ?__init__.py > ? ? ? ?one.py > ? ? ? ?two.py > ? ? ? ?beta/ > ? ? ? ? ? ?__pycache__/ > ? ? ? ? ? ? ? ?__init__.cpython-32.pyc > ? ? ? ? ? ? ? ?__init__.cpython-33.pyc > ? ? ? ? ? ? ? ?__init__.unladen-10.pyc > ? ? ? ? ? ? ? ?three.cpython-32.pyc > ? ? ? ? ? ? ? ?three.cpython-33.pyc > ? ? ? ? ? ? ? ?three.unladen-10.pyc > ? ? ? ? ? ? ? ?four.cpython-32.pyc > ? ? ? ? ? ? ? ?four.cpython-33.pyc > ? ? ? ? ? ? ? ?four.unladen-10.pyc > ? ? ? ? ? ?__init__.py > ? ? ? ? ? ?three.py > ? ? ? ? ? ?four.py > > As you can see, as long as the Python version identifier string is > unique, any number of pyc files can co-exist. ?These identifier > strings are described in more detail below. > > A nice property of this layout is that the `__pycache__` directories > can generally be ignored, such that a normal directory listing would > show something like this:: > > ? ?alpha/ > ? ? ? ?__pycache__/ > ? ? ? ?__init__.py > ? ? ? ?one.py > ? ? ? ?two.py > ? ? ? ?beta/ > ? ? ? ? ? ?__pycache__/ > ? ? ? ? ? ?__init__.py > ? ? ? ? ? ?three.py > ? ? ? ? ? ?four.py > > This is much less cluttered than even today's Python. (Aside: this is a major selling point of this PEP!) > > > Python behavior > =============== > > When Python searches for a module to import (say `foo`), it may find > one of several situations. ?As per current Python rules, the term > "matching pyc" means that the magic number matches the current > interpreter's magic number, and the source file is not newer than the > `pyc` file. > > > Case 1: The first import > ------------------------ > > When Python is asked to import module `foo`, it searches for a > `foo.py` file (or `foo` package, but that's not important for this > discussion) along its `sys.path`. ?When Python locates the `foo.py` > file it will look for a `__pycache__` directory in the directory where > it found the `foo.py`. ?If the `__pycache__` directory is missing, > Python will create it. ?Then it will parse and byte compile the > `foo.py` file and save the byte code in `__pycache__/foo..pyc`, > where is defined by the Python implementation, but will be a > human readable string such as `cpython-32`. (Aside: at first I read this as a description of the full algorithm. But there is a step missing -- the __pycache__/foo..pyc file is searched and not found.) > > > Case 2: The second import > ------------------------- > > When Python is asked to import module `foo` a second time (in a > different process of course), it will again search for the `foo.py` > file along its `sys.path`. ?When Python locates the `foo.py` file, it > looks for a matching `__pycache__/foo..pyc` and finding this, > it reads the byte code and continues as usual. > > > Case 3: __pycache__/foo..pyc with no source > --------------------------------------------------- > > It's possible that the `foo.py` file somehow got removed, while > leaving the cached pyc file still on the file system. ?If the > `__pycache__/foo..pyc` file exists, but the `foo.py` file used > to create it does not, Python will raise an `ImportError` when asked > to import foo. ?In other words, Python will not import a pyc file from > the cache directory unless the source file exists. > > > Case 4: legacy pyc files and source-less imports > ------------------------------------------------ > > Python will ignore all legacy pyc files when a source file exists next > to it. ?In other words, if a `foo.pyc` file exists next to the > `foo.py` file, the pyc file will be ignored in all cases > > In order to continue to support source-less distributions though, if > the source file is missing, Python will import a lone pyc file if it > lives where the source file would have been. > > > Case 5: read-only file systems > ------------------------------ > > When the source lives on a read-only file system, or the `__pycache__` > directory or pyc file cannot otherwise be written, all the same rules > apply. ?This is also the case when `__pycache__` happens to be written > with permissions which do not allow for writing containing pyc files. > > > > Flow chart > ========== > > Here is a flow chart describing how modules are loaded: > > .. image:: pep-3147-1.png > ? :scale: 75 > > > Magic identifiers > ================= > > pyc files inside of the `__pycache__` directories contain a magic > identifier in their file names. ?These are mnemonic tags for the > actual magic numbers used by the importer. ?For example, in Python > 3.2, we could use the hexlified [10]_ magic number as a unique (Aside: when you search Wikipedia for "hexlify" it says "did you mean: heavily?" :-) > identifier:: > > ? ?>>> from binascii import hexlify > ? ?>>> from imp import get_magic > ? ?>>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii')) > ? ?'foo.580c0d0a.pyc' > > This isn't particularly human friendly though. ?Instead, this PEP This section reads a bit weird -- first it describes the solution we *didn't* pick. I'd move that to a "Alternatives Considered and Rejected" section or some such. > proposes a *magic tag* that uniquely defines `.pyc` files for the > current version of Python. ?Whenever the magic number is bumped, a new > magic tag is defined which is unique among all versions and > implementations of Python. ?The actual contents of the magic tag is > left up to the implementation, although it is recommended that the tag > include the implementation name and a version shorthand. ?In general, > magic numbers never change between Python micro releases, but the > convention can be extended to handle magic number changes between > pre-release development versions. > > For example, CPython 3.2 would have a magic tag of `cpython-32` and > write pyc files like this: `foo.cpython-32.pyc`. ?When the `-O` flag > is used, it would write `foo.cpython-32.pyo`. ?For backports of this > feature to Python 2, when the `-U` flag is used, a file such as > `foo.cpython-27u.pyc` can be written. Does all of this match the implementation? > The magic tag is available in the `imp` module via the `get_tag()` > function. ?This is analogous to the `get_magic()` function already > available in that module. > > > Alternative Python implementations > ================================== > > Alternative Python implementations such as Jython [11]_, IronPython > [12]_, PyPy [13]_, Pynie [14]_, and Unladen Swallow can also use the > `__pycache__` directory to store whatever compilation artifacts make > sense for their platforms. ?For example, Jython could store the class > file for the module in `__pycache__/foo.jython-32.class`. > > > Implementation strategy > ======================= > > This feature is targeted for Python 3.2, solving the problem for those > and all future versions. ?It may be back-ported to Python 2.7. Is there time given that 2.7b1 was released? > Vendors are free to backport the changes to earlier distributions as > they see fit. > > > Effects on existing code > ======================== > > Adoption of this PEP will affect existing code and idioms, both inside > Python and outside. ?This section enumerates some of these effects. > > > __file__ > --------- > > In Python 3, when you import a module, its `__file__` attribute points > to its source `py` file (in Python 2, it points to the `pyc` file). ?A > package's `__file__` points to the `py` file for its `__init__.py`. > E.g.:: > > ? ?>>> import foo > ? ?>>> foo.__file__ > ? ?'foo.py' > ? ?# baz is a package > ? ?>>> import baz > ? ?>>> baz.__file__ > ? ?'baz/__init__.py' > > Nothing in this PEP would change the semantics of `__file__`. > > This PEP proposes the addition of an `__cached__` attribute to > modules, which will always point to the actual `pyc` file that was > read or written. ?When the environment variable > `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if > the source lives on a read-only filesystem, then the `__cached__` > attribute will point to the location that the `pyc` file *would* have > been written to if it didn't exist. ?This location of course includes > the `__pycache__` subdirectory in its path. Hm. I wish there was a way to find out whether the bytecode (or whatever) actually *was* read from this file. __file__ in Python 2 supports this (though not in Python 3). > > For alternative Python implementations which do not support `pyc` > files, the `__cached__` attribute may point to whatever information > makes sense. ?E.g. on Jython, this might be the `.class` file for the > module: `__pycache__/foo.jython-32.class`. ?Some implementations may > use multiple compiled files to create the module, in which case > `__cached__` may be a tuple. ?The exact contents of `__cached__` are > Python implementation specific. > > It is recommended that when nothing sensible can be calculated, > implementations should set the `__cached__` attribute to `None`. > > > py_compile and compileall > ------------------------- > > Python comes with two modules, `py_compile` [15]_ and `compileall` > [16]_ which support compiling Python modules external to the built-in > import machinery. ?`py_compile` in particular has intimate knowledge > of byte compilation, so these will be updated to understand the new > layout. ?The `-b` flag is added to `compileall` for writing legacy > `.pyc` byte-compiled file path names. > > > bdist_wininst and the Windows installer > --------------------------------------- > > These tools also compile modules explicitly on installation. ?If they > do not use `py_compile` and `compileall`, then they would also have to > be modified to understand the new layout. > > > File extension checks > --------------------- > > There exists some code which checks for files ending in `.pyc` and > simply chops off the last character to find the matching `.py` file. > This code will obviously fail once this PEP is implemented. > > To support this use case, we'll add two new methods to the `imp` > package [17]_: > > ?* `imp.source_from_cache(py_path)` -> `pyc_path` > ?* `imp.cache_from_source(pyc_path)` -> `py_path` > > Alternative implementations are free to override these functions to > return reasonable values based on their own support for this PEP. > > > PEP 302 loaders > --------------- > > PEP 302 [18]_ defined loaders have a `.get_filename()` method which > points to the `__file__` for a module. ?As part of this PEP, we will > extend this API, to include a new method `.get_paths()` which will > return a 2-tuple containing the path to the source file and the path > to where the matching `pyc` file is (or would be). > > > Backports > --------- > > For versions of Python earlier than 3.2 (and possibly 2.7), it is > possible to backport this PEP. ?However, in Python 3.2 (and possibly > 2.7), this behavior will be turned on by default, and in fact, it will > replace the old behavior. ?Backports will need to support the old > layout by default. ?We suggest supporting PEP 3147 through the use of > an environment variable called `$PYTHONENABLECACHEDIR` or the command > line switch `-Xenablecachedir` to enable the feature. I would be okay if a distro decided to turn it on by default, as long as there was a way to opt out. > > > Makefiles and other dependency tools > ------------------------------------ > > Makefiles and other tools which calculate dependencies on `.pyc` files > (e.g. to byte-compile the source if the `.pyc` is missing) will have > to be updated to check the new paths. > > > Alternatives > ============ > > PEP 304 > ------- > > There is some overlap between the goals of this PEP and PEP 304 [19]_, > which has been withdrawn. ?However PEP 304 would allow a user to > create a shadow file system hierarchy in which to store `pyc` files. > This concept of a shadow hierarchy for `pyc` files could be used to > satisfy the aims of this PEP. ?Although the PEP 304 does not indicate > why it was withdrawn, shadow directories have a number of problems. > The location of the shadow `pyc` files would not be easily discovered > and would depend on the proper and consistent use of the > `$PYTHONBYTECODE` environment variable both by the system and by end > users. ?There are also global implications, meaning that while the > system might want to shadow `pyc` files, users might not want to, but > the PEP defines only an all-or-nothing approach. > > As an example of the problem, a common (though fragile) Python idiom > for locating data files is to do something like this:: > > ? ?from os import dirname, join > ? ?import foo.bar > ? ?data_file = join(dirname(foo.bar.__file__), 'my.dat') > > This would be problematic since `foo.bar.__file__` will give the > location of the `pyc` file in the shadow directory, and it may not be > possible to find the `my.dat` file relative to the source directory > from there. > > > Fat byte compilation files > -------------------------- > > An earlier version of this PEP described "fat" Python byte code files. > These files would contain the equivalent of multiple `pyc` files in a > single `pyf` file, with a lookup table keyed off the appropriate magic > number. ?This was an extensible file format so that the first 5 > parallel Python implementations could be supported fairly efficiently, > but with extension lookup tables available to scale `pyf` byte code > objects as large as necessary. > > The fat byte compilation files were fairly complex, and inherently > introduced difficult race conditions, so the current simplification of > using directories was suggested. ?The same problem applies to using > zip files as the fat pyc file format. > > > Multiple file extensions > ------------------------ > > The PEP author also considered an approach where multiple thin byte > compiled files lived in the same place, but used different file > extensions to designate the Python version. ?E.g. foo.pyc25, > foo.pyc26, foo.pyc31 etc. ?This was rejected because of the clutter > involved in writing so many different files. ?The multiple extension > approach makes it more difficult (and an ongoing task) to update any > tools that are dependent on the file extension. > > > .pyc > ---- > > A proposal was floated to call the `__pycache__` directory `.pyc` or > some other dot-file name. ?This would have the effect on *nix systems > of hiding the directory. ?There are many reasons why this was > rejected by the BDFL [20]_ including the fact that dot-files are only > special on some platforms, and we actually do *not* want to hide these > completely from users. > > > Reference implementation > ======================== > > Work on this code is tracked in a Bazaar branch on Launchpad [22]_ > until it's ready for merge into Python 3.2. ?The work-in-progress diff > can also be viewed [23]_ and is updated automatically as new changes > are uploaded. > > A Rietveld code review issue [24]_ has been opened as of 2010-04-01 (no, > this is not an April Fools joke :). > > > References > ========== > > .. [1] PEP 3146 > > .. [2] The marshal module: > ? http://www.python.org/doc/current/library/marshal.html > > .. [3] import.c: > ? http://svn.python.org/view/python/branches/py3k/Python/import.c?view=markup > > .. [4] Ubuntu: > > .. [5] Debian: > > .. [6] Debian Python Policy: > ? http://www.debian.org/doc/packaging-manuals/python-policy/ > > .. [7] PEP 384 > > .. [8] python-support: > ? http://wiki.debian.org/DebianPythonFAQ#Whatispython-support.3F > > .. [9] python-central: > ? http://wiki.debian.org/DebianPythonFAQ#Whatispython-central.3F > > .. [10] binascii.hexlify(): > ? http://www.python.org/doc/current/library/binascii.html#binascii.hexlify > > .. [11] Jython: http://www.jython.org/ > > .. [12] IronPython: http://ironpython.net/ > > .. [13] PyPy: http://codespeak.net/pypy/dist/pypy/doc/ > > .. [14] Pynie: http://code.google.com/p/pynie/ > > .. [15] py_compile: http://docs.python.org/library/py_compile.html > > .. [16] compileall: http://docs.python.org/library/compileall.html > > .. [17] imp: http://www.python.org/doc/current/library/imp.html > > .. [18] PEP 302 > > .. [19] PEP 304 > > .. [20] http://www.mail-archive.com/python-dev at python.org/msg45203.html > > .. [21] importlib: http://docs.python.org/3.1/library/importlib.html > > .. [22] https://code.launchpad.net/~barry/python/pep3147 > > .. [23] https://code.launchpad.net/~barry/python/pep3147/+merge/22648 > > .. [24] http://codereview.appspot.com/842043/show > > > ACKNOWLEDGMENTS > =============== > > Barry Warsaw's original idea was for fat Python byte code files. > Martin von Loewis reviewed an early draft of the PEP and suggested the > simplification to store traditional `pyc` and `pyo` files in a > directory. ?Many other people reviewed early versions of this PEP and > provided useful feedback including but not limited to: > > * David Malcolm > * Josselin Mouette > * Matthias Klose > * Michael Hudson > * Michael Vogt > * Piotr O?arowski > * Scott Kitterman > * Toshio Kuratomi > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > ? Local Variables: > ? mode: indented-text > ? indent-tabs-mode: nil > ? sentence-end-double-space: t > ? fill-column: 70 > ? coding: utf-8 > ? End: > -- --Guido van Rossum (python.org/~guido) From steve at holdenweb.com Fri Apr 16 06:10:08 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:10:08 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <4BC7E320.5020402@holdenweb.com> Guido van Rossum wrote: [...] >> Implementation strategy >> ======================= >> >> This feature is targeted for Python 3.2, solving the problem for those >> and all future versions. It may be back-ported to Python > > Is there time given that 2.7b1 was released? > I would hope we have learned out lesson about cramming new features in so late in the day, and this *is* a new feature, isn't it? Surely it therefore can't be added in a bugfix release, which in turn means it will never be implemented in Python 2 (given that 2.7 is envisaged as the last Py2 release). >> Vendors are free to backport the changes to earlier distributions as >> they see fit. >> Really? >> >> Effects on existing code >> ======================== >> >> Adoption of this PEP will affect existing code and idioms, both inside >> Python and outside. This section enumerates some of these effects. >> >> >> __file__ >> --------- >> >> In Python 3, when you import a module, its `__file__` attribute points >> to its source `py` file (in Python 2, it points to the `pyc` file). A >> package's `__file__` points to the `py` file for its `__init__.py`. >> E.g.:: >> >> >>> import foo >> >>> foo.__file__ >> 'foo.py' >> # baz is a package >> >>> import baz >> >>> baz.__file__ >> 'baz/__init__.py' >> >> Nothing in this PEP would change the semantics of `__file__`. >> >> This PEP proposes the addition of an `__cached__` attribute to >> modules, which will always point to the actual `pyc` file that was >> read or written. When the environment variable >> `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if >> the source lives on a read-only filesystem, then the `__cached__` >> attribute will point to the location that the `pyc` file *would* have >> been written to if it didn't exist. This location of course includes >> the `__pycache__` subdirectory in its path. > > Hm. I wish there was a way to find out whether the bytecode (or > whatever) actually *was* read from this file. __file__ in Python 2 > supports this (though not in Python 3). > There also seems to be some complexity in this specification. Does the intgerpreter go through the analysis of whether the __pycache__ directory could be created in order to provide the correct value for "the location that the .pyc file would have been written to if it didn't exist"? >> For alternative Python implementations which do not support `pyc` >> files, the `__cached__` attribute may point to whatever information >> makes sense. E.g. on Jython, this might be the `.class` file for the >> module: `__pycache__/foo.jython-32.class`. Some implementations may >> use multiple compiled files to create the module, in which case >> `__cached__` may be a tuple. The exact contents of `__cached__` are >> Python implementation specific. >> >> It is recommended that when nothing sensible can be calculated, >> implementations should set the `__cached__` attribute to `None`. >> >> [...] regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:10:08 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:10:08 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <4BC7E320.5020402@holdenweb.com> Guido van Rossum wrote: [...] >> Implementation strategy >> ======================= >> >> This feature is targeted for Python 3.2, solving the problem for those >> and all future versions. It may be back-ported to Python > > Is there time given that 2.7b1 was released? > I would hope we have learned out lesson about cramming new features in so late in the day, and this *is* a new feature, isn't it? Surely it therefore can't be added in a bugfix release, which in turn means it will never be implemented in Python 2 (given that 2.7 is envisaged as the last Py2 release). >> Vendors are free to backport the changes to earlier distributions as >> they see fit. >> Really? >> >> Effects on existing code >> ======================== >> >> Adoption of this PEP will affect existing code and idioms, both inside >> Python and outside. This section enumerates some of these effects. >> >> >> __file__ >> --------- >> >> In Python 3, when you import a module, its `__file__` attribute points >> to its source `py` file (in Python 2, it points to the `pyc` file). A >> package's `__file__` points to the `py` file for its `__init__.py`. >> E.g.:: >> >> >>> import foo >> >>> foo.__file__ >> 'foo.py' >> # baz is a package >> >>> import baz >> >>> baz.__file__ >> 'baz/__init__.py' >> >> Nothing in this PEP would change the semantics of `__file__`. >> >> This PEP proposes the addition of an `__cached__` attribute to >> modules, which will always point to the actual `pyc` file that was >> read or written. When the environment variable >> `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if >> the source lives on a read-only filesystem, then the `__cached__` >> attribute will point to the location that the `pyc` file *would* have >> been written to if it didn't exist. This location of course includes >> the `__pycache__` subdirectory in its path. > > Hm. I wish there was a way to find out whether the bytecode (or > whatever) actually *was* read from this file. __file__ in Python 2 > supports this (though not in Python 3). > There also seems to be some complexity in this specification. Does the intgerpreter go through the analysis of whether the __pycache__ directory could be created in order to provide the correct value for "the location that the .pyc file would have been written to if it didn't exist"? >> For alternative Python implementations which do not support `pyc` >> files, the `__cached__` attribute may point to whatever information >> makes sense. E.g. on Jython, this might be the `.class` file for the >> module: `__pycache__/foo.jython-32.class`. Some implementations may >> use multiple compiled files to create the module, in which case >> `__cached__` may be a tuple. The exact contents of `__cached__` are >> Python implementation specific. >> >> It is recommended that when nothing sensible can be calculated, >> implementations should set the `__cached__` attribute to `None`. >> >> [...] regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:13:57 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:13:57 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: <4BC7E405.9080304@holdenweb.com> Brian Curtin wrote: > On Thu, Apr 15, 2010 at 03:20, anatoly techtonik > wrote: > > On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan > wrote: > > > >>> I am surprised to see that the bug-tracker > >>> doesn't have an OS classifier or ability to add > >>> tags ? Since a number of issues reported seem to > > Just to remind about my +1 for user editable tags. > > > -sys.maxint on just about anything user editable except for the title or > messages on an issue. I don't think user editable tags on an issue bring > anything to the table except making it seem more "web 2.0". Searching > user-generated tags would be a nightmare because you'd have to know > every combination of some idea in order to get relevant results (e.g., > windows, windoze, window$, win32). > Yes, tight vocabulary control will lead to more consistent classifications. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:13:57 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:13:57 -0400 Subject: [Python-Dev] OS information, tags In-Reply-To: References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> Message-ID: <4BC7E405.9080304@holdenweb.com> Brian Curtin wrote: > On Thu, Apr 15, 2010 at 03:20, anatoly techtonik > wrote: > > On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan > wrote: > > > >>> I am surprised to see that the bug-tracker > >>> doesn't have an OS classifier or ability to add > >>> tags ? Since a number of issues reported seem to > > Just to remind about my +1 for user editable tags. > > > -sys.maxint on just about anything user editable except for the title or > messages on an issue. I don't think user editable tags on an issue bring > anything to the table except making it seem more "web 2.0". Searching > user-generated tags would be a nightmare because you'd have to know > every combination of some idea in order to get relevant results (e.g., > windows, windoze, window$, win32). > Yes, tight vocabulary control will lead to more consistent classifications. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:17:39 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:17:39 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC6142B.7000002@v.loewis.de> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> <4BC6142B.7000002@v.loewis.de> Message-ID: <4BC7E4E3.4060808@holdenweb.com> Martin v. L?wis wrote: >> Michael Foord wrote: >> >>> Building the Mac installer requires volunteer time which I'm not sure >>> that more hardware will fix - compiling a full build of Python for Mac >>> OS X (with all the Python modules like Tkinter etc) requires expertise >>> which only a few people have. >> That's nuts. Why isn't this expertise captured in the form of a script? > > Much of it is, but it still takes expertise to run the script. > > It would take even more expertise to capture the remaining pieces in the > script, too, and no living person has that much expertise to write the > script (perhaps there are one or two people, and they don't have the time). > I take it you don't mean to imply that there's a dead person somewhere with the necessary expertise? [ducks]. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:17:39 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:17:39 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC6142B.7000002@v.loewis.de> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> <4BC6142B.7000002@v.loewis.de> Message-ID: <4BC7E4E3.4060808@holdenweb.com> Martin v. L?wis wrote: >> Michael Foord wrote: >> >>> Building the Mac installer requires volunteer time which I'm not sure >>> that more hardware will fix - compiling a full build of Python for Mac >>> OS X (with all the Python modules like Tkinter etc) requires expertise >>> which only a few people have. >> That's nuts. Why isn't this expertise captured in the form of a script? > > Much of it is, but it still takes expertise to run the script. > > It would take even more expertise to capture the remaining pieces in the > script, too, and no living person has that much expertise to write the > script (perhaps there are one or two people, and they don't have the time). > I take it you don't mean to imply that there's a dead person somewhere with the necessary expertise? [ducks]. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:18:30 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:18:30 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <49589.1271279777@parc.com> References: <4BA62546.10906@python.org> <4BC58326.4@voidspace.org.uk> <41372.1271259395@parc.com> <4BC6142B.7000002@v.loewis.de> <49589.1271279777@parc.com> Message-ID: Bill Janssen wrote: > Martin v. L?wis wrote: [..] >> It would take even more expertise to capture the remaining pieces in the >> script, too, and no living person has that much expertise to write the >> script (perhaps there are one or two people, and they don't have the time). > > Well, God forbid they should ever be hit by a bus! This kind of thing > needs to be written down. > +1 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:36:55 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:36:55 -0400 Subject: [Python-Dev] Request for commit access In-Reply-To: <4BB04655.6060008@v.loewis.de> References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> <4BB04655.6060008@v.loewis.de> Message-ID: <4BC7E967.2080506@holdenweb.com> Martin v. L?wis wrote: >> That's an ideal case, but it doesn't work, because more or less senior >> committers are already too busy. If they do not even have time to >> review issues, followup on patches - how can they monitor who reached >> the appropriate karma level? > > The practice proves to be different. In the recent additions, the new > developer had been approached by some committer, suggesting that they > apply. So the initiative actually started from a committer who thought > that the karma level was appropriate. All I'm asking for is that then > this committer takes the issues in his own hands, rather than letting > the new developer ask for himself. > That sounds like it should be the preferred procedure, but it may be that Brian was advised to ask by an existing committer who didn't feel they had the "authority" to recommend him. In such an event (or if committers are too busy to action a promising candidate) I don't see anything wrong with a candidate asking directly (as long as they are prepared to put up with a semi-public discussion of their application's merits). regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 06:36:55 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 00:36:55 -0400 Subject: [Python-Dev] Request for commit access In-Reply-To: <4BB04655.6060008@v.loewis.de> References: <20100323152748.50ECE1FD33F@kimball.webabinitio.net> <4BA94202.6060502@v.loewis.de> <4BB04655.6060008@v.loewis.de> Message-ID: <4BC7E967.2080506@holdenweb.com> Martin v. L?wis wrote: >> That's an ideal case, but it doesn't work, because more or less senior >> committers are already too busy. If they do not even have time to >> review issues, followup on patches - how can they monitor who reached >> the appropriate karma level? > > The practice proves to be different. In the recent additions, the new > developer had been approached by some committer, suggesting that they > apply. So the initiative actually started from a committer who thought > that the karma level was appropriate. All I'm asking for is that then > this committer takes the issues in his own hands, rather than letting > the new developer ask for himself. > That sounds like it should be the preferred procedure, but it may be that Brian was advised to ask by an existing committer who didn't feel they had the "authority" to recommend him. In such an event (or if committers are too busy to action a promising candidate) I don't see anything wrong with a candidate asking directly (as long as they are prepared to put up with a semi-public discussion of their application's merits). regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 07:30:13 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 01:30:13 -0400 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) In-Reply-To: References: Message-ID: <4BC7F5E5.7030203@holdenweb.com> anatoly techtonik wrote: > On Mon, Mar 29, 2010 at 4:50 PM, Tarek Ziad? wrote: >> Anatoly, I am now answering only in Distutils-SIG. >> >> On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik wrote: >> [..] > > Seems like I start to hate mailing lists even more with all this > message duplication and thread following nightmare. Why can't people > here create a Google Groups mirror? > I would surmise that most people on python-dev do what I do and subscribe to an NNTP (Usenet news) source. In my case Thunderbird does a great job of presenting the content, though there is no thread subscription. Google Groups have their disadvantages too. I personally dislike "web forum" style interfaces. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 16 07:30:13 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 01:30:13 -0400 Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for package management tool in Python 2.7) In-Reply-To: References: Message-ID: <4BC7F5E5.7030203@holdenweb.com> anatoly techtonik wrote: > On Mon, Mar 29, 2010 at 4:50 PM, Tarek Ziad? wrote: >> Anatoly, I am now answering only in Distutils-SIG. >> >> On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik wrote: >> [..] > > Seems like I start to hate mailing lists even more with all this > message duplication and thread following nightmare. Why can't people > here create a Google Groups mirror? > I would surmise that most people on python-dev do what I do and subscribe to an NNTP (Usenet news) source. In my case Thunderbird does a great job of presenting the content, though there is no thread subscription. Google Groups have their disadvantages too. I personally dislike "web forum" style interfaces. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From martin at v.loewis.de Fri Apr 16 07:35:06 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 16 Apr 2010 07:35:06 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> Message-ID: <4BC7F70A.9000503@v.loewis.de> > Keeping the knowledge in the makefile or script in the source tree would > let the rules change across branches without affecting the build master. > Though if having more specific rules in the master was easier, I'd be > fine with that too. Intuitively, I find the set of batch files used for Windows "ugly" (despite me having created them in the first place). Objectively, I think that has the risk of getting "out of hands", in the sense that people start to think that they have to use these scripts in order to build Python on Mac. This actually happened on Windows - some people now recommend to run the buildbot scripts on a regular developer checkout, because they supposedly do the right things. I would rather prefer to have the buildbot run the same commands that we recommend developers to run. The knowledge about these should be in the README, and it should be stable knowledge, i.e. require infrequent changes. This is true for Unix: configure/make/make test/make clean had been the correct procedure for ten years now. The Unix buildbot only deviates slightly, to have the slaves run a more expensive version of the test suite. > Agreed, thus my caveat as to the output of the build-installer script > in fact being what has been published for downloads. In my brief > tests it looks like it created a production DMG ready for publishing > on the download page, and did in fact install correctly, but I'm still > new to the Mac binary building process. I'd be interested in setting up daily builds then. For these, I think it's fine that they may differ slightly from the official binaries - people would recognize that they are testing a different set of binaries. > Of course, this is just the DMG construction. Not sure what amount of > "test the installer" testing should be required prior to publication. If it's fully automated, the amount of testing may actually be small, IMO. I personally would put timeliness over correctness here. Also, it might be that we can sign up some volunteer to do a smoke test in a timely manner at release time who will test the installer on, say, a clean VM. Regards, Martin From alex.gaynor at gmail.com Fri Apr 16 06:57:06 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 16 Apr 2010 00:57:06 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior Message-ID: Hi all, I ran into the follow behavior while making sure Django works correctly on PyPy. The following behavior was observed in all tested versions of CPython (2.5, 3.1): >>> def f(**kwargs): ... print(kwargs) ... >>> kwargs = {1: 3} >>> >>> dict({}, **kwargs) {1: 3} >>> f(**kwargs) Traceback (most recent call last): File "", line 1, in TypeError: f() keywords must be strings >>> This behavior seems pretty strange to me, indeed PyPy gives the TypeError for both attempts. I just wanted to confirm that it was in fact intentional. Thanks, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me From phd at phd.pp.ru Fri Apr 16 09:31:31 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 16 Apr 2010 11:31:31 +0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: Message-ID: <20100416073131.GA21438@phd.pp.ru> On Fri, Apr 16, 2010 at 12:57:06AM -0400, Alex Gaynor wrote: > >>> def f(**kwargs): > ... print(kwargs) > ... > >>> kwargs = {1: 3} > >>> > >>> dict({}, **kwargs) > {1: 3} > >>> f(**kwargs) > Traceback (most recent call last): > File "", line 1, in > TypeError: f() keywords must be strings Argument names must be strings. In your example 1 must be at least '1'. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From hagen at zhuliguan.net Fri Apr 16 10:04:38 2010 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Fri, 16 Apr 2010 10:04:38 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: Message-ID: <4BC81A16.6050107@zhuliguan.net> > This behavior seems pretty strange to me, indeed PyPy gives the > TypeError for both attempts. I just wanted to confirm that it was in > fact intentional. Oleg already answered why f(**{1:3}) raises a TypeError. But your question seems to be rather why dict(**{1:3}) doesn't. For functions implemented in Python, non-string arguments are always rejected, but C functions (like the dict constructor) don't have to reject them. I don't see any benefit in allowing them, but it's probably not worth breaking code by disallowing them either. I couldn't find this documented. Perhaps we should just say "don't rely on being able to pass non-string keywords" somewhere? - Hagen From dickinsm at gmail.com Fri Apr 16 10:29:21 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 16 Apr 2010 09:29:21 +0100 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <4BC81A16.6050107@zhuliguan.net> References: <4BC81A16.6050107@zhuliguan.net> Message-ID: On Fri, Apr 16, 2010 at 9:04 AM, Hagen F?rstenau wrote: >> This behavior seems pretty strange to me, indeed PyPy gives the >> TypeError for both attempts. ?I just wanted to confirm that it was in >> fact intentional. > > Oleg already answered why f(**{1:3}) raises a TypeError. But your > question seems to be rather why dict(**{1:3}) doesn't. > > For functions implemented in Python, non-string arguments are always > rejected, but C functions (like the dict constructor) don't have to > reject them. I don't see any benefit in allowing them, but it's probably > not worth breaking code by disallowing them either. "dict(x, **y)" as an expression version of x.update(y) seems to be fairly well known[1], so disallowing non-string keyword arguments seems likely to break existing code, as well as (probably?) harming performance. So I can't see CPython changing here. I'm not sure whether other implementations should be required to follow suit, though---maybe this should be regarded as an implementation-defined detail? Mark [1] http://stackoverflow.com/questions/38987/how-can-i-merge-two-python-dictionaries-as-a-single-expression ) From solipsis at pitrou.net Fri Apr 16 13:03:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 16 Apr 2010 11:03:30 +0000 (UTC) Subject: [Python-Dev] urlparse - to parse IPv6 URL in 2.7 b2 ? References: <20100415160718.GA12597@remy> Message-ID: Senthil Kumaran gmail.com> writes: > > http://bugs.python.org/issue2987 > > This deals with a feature request of parsing an IPv6 URL according to > standards. The patch is pretty complete and we have good test coverage > too. > > Is it okay to include this in Python 2.7 b2 release? It would be a > worthy addition. > It shouldn't have been committed to 3.1, though. Could you revert? Thanks Antoine. From orsenthil at gmail.com Fri Apr 16 13:15:01 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Fri, 16 Apr 2010 16:45:01 +0530 Subject: [Python-Dev] urlparse - to parse IPv6 URL in 2.7 b2 ? In-Reply-To: References: <20100415160718.GA12597@remy> Message-ID: <20100416111501.GB3712@remy> On Fri, Apr 16, 2010 at 11:03:30AM +0000, Antoine Pitrou wrote: > > It shouldn't have been committed to 3.1, though. Could you revert? Yeah, I had this doubt. Okay, I shall revert it from 3.1 branch. -- Senthil From brian.curtin at gmail.com Fri Apr 16 15:01:54 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 16 Apr 2010 08:01:54 -0500 Subject: [Python-Dev] MSDN licenses available for python-dev Message-ID: Hi python-dev, The recent threads on builds/installers for Mac and Windows reminded me of Steve Holden's push to get the python-dev team equipped via a connection with the Microsoft Open Source Technology Center. The OSTC team provides Microsoft Developer Network licenses to open source projects to assist them in better supporting Windows. I've talked with Steve (who passed the task to me) and the Microsoft folks, and they are happy to provide more licenses if needed. If you are interested in getting a copy of MSDN, please contact me off-list. I'll provide you with a code that you'll put into their site, then around a week later you should get your subscription. The snippet below is taken from prior correspondence with the OSTC team in regards to who can receive the licenses: """ For the purposes of providing MSDN licenses to an open source development community, I consider anyone who writes, builds, tests or documents software to be a "developer who contributes" to the project. (In fact, having started out as a test engineer, I would take exception to anyone who claimed only people who write code are "developers" :-) We do ask that requests are for people who are active contributors and not just minor/occasional participants. """ If this applies to you and you are interested, let me know. Brian Curtin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Apr 16 16:15:59 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Apr 2010 00:15:59 +1000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> Message-ID: <4BC8711F.8060001@gmail.com> Mark Dickinson wrote: > "dict(x, **y)" as an expression version of x.update(y) seems to be > fairly well known[1], so disallowing non-string keyword arguments > seems likely to break existing code, as well as (probably?) harming > performance. So I can't see CPython changing here. I'm not sure > whether other implementations should be required to follow suit, > though---maybe this should be regarded as an implementation-defined > detail? I would agree with leaving it implementation defined - I don't think either PyPy or CPython should be forced to change their current behaviour in relation to this. A minor note in the language reference to that effect may be worthwhile just to make that stance official. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Fri Apr 16 16:33:28 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 07:33:28 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <4BC8711F.8060001@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 7:15 AM, Nick Coghlan wrote: > Mark Dickinson wrote: >> "dict(x, **y)" as an expression version of x.update(y) seems to be >> fairly well known[1], so disallowing non-string keyword arguments >> seems likely to break existing code, as well as (probably?) harming >> performance. ?So I can't see CPython changing here. ?I'm not sure >> whether other implementations should be required to follow suit, >> though---maybe this should be regarded as an implementation-defined >> detail? > > I would agree with leaving it implementation defined - I don't think > either PyPy or CPython should be forced to change their current > behaviour in relation to this. A minor note in the language reference to > that effect may be worthwhile just to make that stance official. That is just going to cause some programs to have a portability surprise. I think one or the other should be fixed. I am fine with declaring dict({}, **{1:3}) illegal, since after all it is abuse of the ** mechanism. We should deprecate it in at least one version though. -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Fri Apr 16 16:47:20 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 16 Apr 2010 15:47:20 +0100 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 3:33 PM, Guido van Rossum wrote: > On Fri, Apr 16, 2010 at 7:15 AM, Nick Coghlan wrote: >> I would agree with leaving it implementation defined - I don't think >> either PyPy or CPython should be forced to change their current >> behaviour in relation to this. A minor note in the language reference to >> that effect may be worthwhile just to make that stance official. > > That is just going to cause some programs to have a portability > surprise. I think one or the other should be fixed. I am fine with > declaring dict({}, **{1:3}) illegal, since after all it is abuse of > the ** mechanism. We should deprecate it in at least one version > though. Okay; I'll open an issue for deprecation in 3.2 and removal in 3.3. Can this sneak in under the 'incorrect language semantics' exemption for PEP 3003 (the moratorium PEP)? If not, then deprecation presumably has to wait for 3.3. Mark From draghuram at gmail.com Fri Apr 16 16:49:35 2010 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Fri, 16 Apr 2010 10:49:35 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: Message-ID: On Fri, Apr 16, 2010 at 12:57 AM, Alex Gaynor wrote: > Hi all, > > I ran into the follow behavior while making sure Django works > correctly on PyPy. ?The following behavior was observed in all tested > versions of CPython (2.5, 3.1): > >>>> def f(**kwargs): > ... ? ? print(kwargs) > ... >>>> kwargs = {1: 3} >>>> >>>> dict({}, **kwargs) > {1: 3} >>>> f(**kwargs) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: f() keywords must be strings >>>> > > > This behavior seems pretty strange to me, indeed PyPy gives the > TypeError for both attempts. ?I just wanted to confirm that it was in > fact intentional. I ran into same issue with Django on Jython yesterday [1] since Jython too gives TypeError for 'dict({}, **kwargs)'. Thanks, Raghu [1] http://bugs.jython.org/issue1600 From solipsis at pitrou.net Fri Apr 16 16:57:38 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 16 Apr 2010 14:57:38 +0000 (UTC) Subject: [Python-Dev] Very Strange Argument Handling Behavior References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > Okay; I'll open an issue for deprecation in 3.2 and removal in 3.3. > > Can this sneak in under the 'incorrect language semantics' exemption > for PEP 3003 (the moratorium PEP)? If not, then deprecation > presumably has to wait for 3.3. It seems that in spirit the moratorium applies more to language additions than to removals/limitations. The goal being that alternate implementation stop chasing a moving target in terms of features. So IMVHO it is fine for 3.2. Regards Antoine. From dickinsm at gmail.com Fri Apr 16 17:06:38 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 16 Apr 2010 16:06:38 +0100 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 3:57 PM, Antoine Pitrou wrote: > Mark Dickinson gmail.com> writes: >> >> Okay; ?I'll open an issue for deprecation in 3.2 and removal in 3.3. >> >> Can this sneak in under the 'incorrect language semantics' exemption >> for PEP 3003 (the moratorium PEP)? ?If not, then deprecation >> presumably has to wait for 3.3. > > It seems that in spirit the moratorium applies more to language additions than > to removals/limitations. The goal being that alternate implementation stop > chasing a moving target in terms of features. > > So IMVHO it is fine for 3.2. Removing it certainly seems in keeping with the goal of making life easier for alternate implementations. (Out of curiosity, does anyone know what IronPython does here?) I've opened http://bugs.python.org/issue8419 Mark From fuzzyman at voidspace.org.uk Fri Apr 16 17:17:28 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 16 Apr 2010 17:17:28 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: <4BC87F88.705@voidspace.org.uk> On 16/04/2010 17:06, Mark Dickinson wrote: > On Fri, Apr 16, 2010 at 3:57 PM, Antoine Pitrou wrote: > >> Mark Dickinson gmail.com> writes: >> >>> Okay; I'll open an issue for deprecation in 3.2 and removal in 3.3. >>> >>> Can this sneak in under the 'incorrect language semantics' exemption >>> for PEP 3003 (the moratorium PEP)? If not, then deprecation >>> presumably has to wait for 3.3. >>> >> It seems that in spirit the moratorium applies more to language additions than >> to removals/limitations. The goal being that alternate implementation stop >> chasing a moving target in terms of features. >> >> So IMVHO it is fine for 3.2. >> > Removing it certainly seems in keeping with the goal of making life > easier for alternate implementations. (Out of curiosity, does anyone > know what IronPython does here?) > Same as Jython and PyPy: IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.4927 Type "help", "copyright", "credits" or "license" for more information. >>> dict(**{1: 'hi'}) Traceback (most recent call last): File "", line 1, in TypeError: expected string for dictionary argument got 1 >>> Michael > I've opened http://bugs.python.org/issue8419 > > Mark > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From ronaldoussoren at mac.com Fri Apr 16 17:22:06 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 16 Apr 2010 17:22:06 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: <21741669123421670525391446873886509887-Webmail@me.com> On Friday, April 16, 2010, at 04:57PM, "Antoine Pitrou" wrote: >Mark Dickinson gmail.com> writes: >> >> Okay; I'll open an issue for deprecation in 3.2 and removal in 3.3. >> >> Can this sneak in under the 'incorrect language semantics' exemption >> for PEP 3003 (the moratorium PEP)? If not, then deprecation >> presumably has to wait for 3.3. > >It seems that in spirit the moratorium applies more to language additions than >to removals/limitations. The goal being that alternate implementation stop >chasing a moving target in terms of features. > >So IMVHO it is fine for 3.2. What about 2.7, should it be deprecated there as well? Ronald From dinov at microsoft.com Fri Apr 16 17:22:34 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 16 Apr 2010 15:22:34 +0000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: <1A472770E042064698CB5ADC83A12ACD39532BE6@TK5EX14MBXC118.redmond.corp.microsoft.com> Mark Dickinson wrote: > Removing it certainly seems in keeping with the goal of making life > easier for alternate implementations. (Out of curiosity, does anyone > know what IronPython does here?) > > I've opened http://bugs.python.org/issue8419 It looks like IronPython reports a type error as well: IronPython 2.6.1 DEBUG (2.6.10920.0) on .NET 2.0.50727.4927 Type "help", "copyright", "credits" or "license" for more information. >>> def f(**kwargs): ... print(kwargs) ... >>> kwargs = {1: 3} >>> >>> dict({}, **kwargs) Traceback (most recent call last): File "", line unknown, in TypeError: expected string for dictionary argument got 1 >>> d = {1:2} >>> d.update({3:4}, **{5:6}) Traceback (most recent call last): File "", line unknown, in TypeError: expected string for dictionary argument got 5 From stefan_ml at behnel.de Fri Apr 16 17:35:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 16 Apr 2010 17:35:56 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: Guido van Rossum, 16.04.2010 16:33: > I am fine with > declaring dict({}, **{1:3}) illegal, since after all it is abuse of > the ** mechanism. It's a bit like letting keys like 'not-an-identifier' pass through, though, isn't it? Stefan From barry at python.org Fri Apr 16 18:02:20 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 16 Apr 2010 12:02:20 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <20100416120220.50cf21a9@heresy> On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >Comments inline. Nothing showstopping, mostly just spewing obscure >background information... > >Overall, congratulations! I'm fine with the implementation going in >and the PEP being marked as accepted as long as you get to the >clarifications I suggest below soon after. Awesome, thanks Guido! I will respond in detail and address your clarifications before I commit to the py3k branch. I wanted to address one thing now though since Steve responded to it. >> Implementation strategy >> ======================= >> >> This feature is targeted for Python 3.2, solving the problem for those >> and all future versions. ?It may be back-ported to Python 2.7. > >Is there time given that 2.7b1 was released? I think this would be totally up to Benjamin as the RM for 2.7. Although I haven't tried yet, my sense of it is that most of the patch would port pretty easily to trunk. I could probably generate a patch for review by mid-next week. Whether it should or not is a different matter. Given that we're in beta, I'm not sure *I* would approve it if I were the RM, but as the developer, sure, I'd love to back port it. :) However... >> Vendors are free to backport the changes to earlier distributions as >> they see fit. ...Steve asks if we're really going to do this. For Debian and/or Ubuntu, we haven't yet decided. I plan to begin that discussion on the appropriate distro-related mailing lists after the code lands on py3k. It certainly won't be enabled by default. I don't think we've made any decisions about which versions of Python will make it into the next release of Ubuntu about 6 months from now, but given the Python release schedule, that could be 2.6, 2.7 and 3.1. If we really do include all three versions, I will push for backporting the feature (enabled with -Xcachedir) in our releases so that we can gain the benefit of ditching the symlink farms as soon as possible. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From status at bugs.python.org Fri Apr 16 18:07:34 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 16 Apr 2010 18:07:34 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100416160734.6B6CC78129@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-04-09 - 2010-04-16) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2658 open (+38) / 17582 closed (+25) / 20240 total (+63) Open issues with patches: 1090 Average duration of open issues: 728 days. Median duration of open issues: 492 days. Open Issues Breakdown open 2616 (+38) languishing 9 ( +0) pending 32 ( +0) Issues Created Or Reopened (68) _______________________________ Add "key" argument to "bisect" module functions 2010-04-14 http://bugs.python.org/issue4356 reopened rhettinger patch AttributeError: MSVCCompiler instance has no attribute '_MSVCC 2010-04-13 http://bugs.python.org/issue7509 reopened tarek configure: ignore AC_PROG_CC hardcoded CFLAGS 2010-04-11 http://bugs.python.org/issue8211 reopened lemburg patch test_ctypes fails in test_ulonglong on sparc buildbots 2010-04-12 http://bugs.python.org/issue8314 reopened pitrou patch, buildbot Add a --show-installation-paths in the install command 2010-04-09 http://bugs.python.org/issue8357 reopened tarek absolute_import future directive ignored by 2to3 2010-04-09 CLOSED http://bugs.python.org/issue8358 created jaraco % formating - TypeError: not all arguments converted during st 2010-04-10 CLOSED http://bugs.python.org/issue8359 created sneetsher doc: unittest.skipTest method added in 2.7 2010-04-10 CLOSED http://bugs.python.org/issue8360 created techtonik patch Remove assert in functools.total_ordering 2010-04-10 CLOSED http://bugs.python.org/issue8361 created merwok patch Add Misc/maintainers.rst to 2.x branch 2010-04-10 http://bugs.python.org/issue8362 created merwok patch, needs review Lots of duplicate code between test_classify_oldstyle and test 2010-04-10 CLOSED http://bugs.python.org/issue8363 created exarkun patch, needs review Update site.setquit docstring 2010-04-10 CLOSED http://bugs.python.org/issue8364 created merwok patch 'readline' module fails to build on OS X - some recent change 2010-04-11 CLOSED http://bugs.python.org/issue8365 created l0nwlf OS X universal builds fail on 2.7b1 and py3k with "Don't know 2010-04-11 http://bugs.python.org/issue8366 created ned.deily test_winsound: failure on systems without soundcard 2010-04-11 CLOSED http://bugs.python.org/issue8367 created skrah patch Memory leak on multi-threaded PyObject_CallObject 2010-04-11 CLOSED http://bugs.python.org/issue8368 created Krauzi Add a lint command to distutils 2010-04-11 http://bugs.python.org/issue8369 created merwok change module "builtins" to "__builtin__" in __import__ docume 2010-04-11 CLOSED http://bugs.python.org/issue8370 created cjerdonek patch Add a command to download distributions 2010-04-11 http://bugs.python.org/issue8371 created merwok socket: Buffer overrun while reading unterminated AF_UNIX addr 2010-04-11 http://bugs.python.org/issue8372 created baikie patch socket: AF_UNIX socket paths not handled according to PEP 383 2010-04-11 http://bugs.python.org/issue8373 created baikie patch Some locales are unsupported 2010-04-11 CLOSED http://bugs.python.org/issue8374 created ubuntujenkins test_distutils fails when run with -j 2010-04-11 CLOSED http://bugs.python.org/issue8375 created pitrou Tutorial offers dangerous advice about iterators: ???__iter__( 2010-04-12 http://bugs.python.org/issue8376 created andersk Errata on page:http://docs.python.org/library/stdtypes.html 2010-04-12 CLOSED http://bugs.python.org/issue8377 created jatin085 PYTHONSTARTUP is not run by default when Idle is started 2010-04-12 http://bugs.python.org/issue8378 created jvanpraag if __debug__: has nonobvious behaviour when evaluating .pyo wi 2010-04-12 CLOSED http://bugs.python.org/issue8379 created afriesen Port of the gdb7 debugging hooks to the "py3k" branch 2010-04-12 http://bugs.python.org/issue8380 created dmalcolm patch IDLE 2.6 freezes on OS X 10.6 2010-04-12 http://bugs.python.org/issue8381 created aaron.the.cow StringIO.write() takes any argument and converts it to a strin 2010-04-12 CLOSED http://bugs.python.org/issue8382 created tarek pickle is unable to encode unicode surrogates 2010-04-13 CLOSED http://bugs.python.org/issue8383 created haypo patch Distutils C extension build with MinGW on Windows fails 2010-04-13 http://bugs.python.org/issue8384 created cmcqueen1975 easy _winreg remaining in test_winsound 2010-04-13 CLOSED http://bugs.python.org/issue8385 created tim.golden patch test_pickle failing 2010-04-13 CLOSED http://bugs.python.org/issue8386 created tim.golden use universal newline mode in csv module examples 2010-04-13 http://bugs.python.org/issue8387 created sfinnie None shouldn't be passed to traceback.format_exception 2010-04-13 http://bugs.python.org/issue8388 created benjamin.peterson Incomprehensible import error 2010-04-13 CLOSED http://bugs.python.org/issue8389 created torsten tarfile: use surrogates for undecode fields 2010-04-13 http://bugs.python.org/issue8390 created haypo patch os.execvpe() doesn't support surrogates in env 2010-04-14 http://bugs.python.org/issue8391 created haypo patch unit tests rather light on testing __import__(..., level) 2010-04-14 http://bugs.python.org/issue8392 created skip.montanaro patch, needs review subprocess: support undecodable current working directory on P 2010-04-14 http://bugs.python.org/issue8393 created haypo patch ctypes.dlopen() doesn't support surrogates 2010-04-14 http://bugs.python.org/issue8394 created haypo patch shutil.copytree() add a copy_func parameter. 2010-04-14 http://bugs.python.org/issue8395 created ysj.ray tarfile.open does fails with UnicodeDecodeError if parent dir 2010-04-14 CLOSED http://bugs.python.org/issue8396 created parthm BZ2File doesn't protect against mixed iterator and read usage 2010-04-14 http://bugs.python.org/issue8397 created Alex.Stapleton Modules with a dot in the filename don't load 2010-04-14 CLOSED http://bugs.python.org/issue8398 created giunghi Document os.open 2010-04-14 CLOSED http://bugs.python.org/issue8399 reopened techtonik patch zipimporter find_module fullname mis-documented 2010-04-14 http://bugs.python.org/issue8400 created kmtracey Strange behavior of bytearray slice assignment 2010-04-14 http://bugs.python.org/issue8401 created abacabadabacaba patch glob returns empty list with "[" character in the folder name 2010-04-15 http://bugs.python.org/issue8402 reopened george.hu dis.dis gives different results if Ctrl-C is pressed 2010-04-15 http://bugs.python.org/issue8403 created ivank Set operations don't work for dictionary views 2010-04-15 http://bugs.python.org/issue8404 created akuchling patch Improve test_os._kill (failing on slow machines) 2010-04-15 http://bugs.python.org/issue8405 created brian.curtin patch Make some setup.py paths exclude-able 2010-04-15 http://bugs.python.org/issue8406 created skip.montanaro patch expose signalfd(2) and sigprocmask(2) in the signal module 2010-04-15 http://bugs.python.org/issue8407 created exarkun need console/pager module 2010-04-15 CLOSED http://bugs.python.org/issue8408 created techtonik gettext should honor $LOCPATH environment variable 2010-04-15 http://bugs.python.org/issue8409 created barry easy Fix emulated lock to be 'fair' 2010-04-15 http://bugs.python.org/issue8410 created krisvale patch, patch, needs review Improve condition variable emulation on NT 2010-04-15 http://bugs.python.org/issue8411 created krisvale patch, patch, needs review os.system() doesn't support surrogates nor bytes 2010-04-16 CLOSED http://bugs.python.org/issue8412 created haypo patch String interpolation doesn't work with sys.version_info 2010-04-16 http://bugs.python.org/issue8413 created Arfrever Add test cases for assert 2010-04-16 http://bugs.python.org/issue8414 created gnofi patch namedtuple vs tuple 2010-04-16 CLOSED http://bugs.python.org/issue8415 created hkrishnan python 2.6.5 documentation can't search 2010-04-16 http://bugs.python.org/issue8416 created sgala bytes and bytearray constructors raise TypeError for very larg 2010-04-16 http://bugs.python.org/issue8417 created abacabadabacaba Crash 0xc0000417 STATUS_INVALID_CRUNTIME_PARAMETER on program 2010-04-16 http://bugs.python.org/issue8418 created alfps dict constructor allows invalid identifiers in **kwargs 2010-04-16 http://bugs.python.org/issue8419 created mark.dickinson set_lookkey is unsafe 2010-04-16 http://bugs.python.org/issue8420 created abacabadabacaba Issues Now Closed (54) ______________________ _fmode = O_TEXT is obsolete 797 days http://bugs.python.org/issue2028 techtonik add ftp-tls support to ftplib - RFC 4217 793 days http://bugs.python.org/issue2054 orsenthil patch __import__ with fromlist= 793 days http://bugs.python.org/issue2090 gsakkis RFC2732 support for urlparse (IPv6 addresses) 689 days http://bugs.python.org/issue2987 orsenthil patch, easy failures in test_uuid 603 days http://bugs.python.org/issue3581 skrah patch Build failure on OpenBSD 4.4-current regarding lstat() 486 days http://bugs.python.org/issue4639 skrah patch Document __instancecheck__ and __subclasscheck__ 425 days http://bugs.python.org/issue5250 georg.brandl patch email message.get_params() and related methods sometimes fail. 423 days http://bugs.python.org/issue5277 r.david.murray patch, easy A selection of spelling errors and typos throughout source 69 days http://bugs.python.org/issue5341 merwok patch Strange behavior when performing int on a Decimal made from -s 411 days http://bugs.python.org/issue5377 mark.dickinson patch urllib.request.open(someURL).read() returns a bytes object so 407 days http://bugs.python.org/issue5419 orsenthil easy 2to3 -j 4 generates malformed diffs 283 days http://bugs.python.org/issue6409 benjamin.peterson patch test_urllib: unsetting missing 'env' variable 20 days http://bugs.python.org/issue7026 orsenthil Add environment variable $PYTHONWARNINGS 8 days http://bugs.python.org/issue7301 pjenvey patch, needs review Patch - skip winsound tests if no default sound is configured 152 days http://bugs.python.org/issue7306 brian.curtin patch, needs review Add a timeout functionality to common locking operations 152 days http://bugs.python.org/issue7316 torsten patch, needs review difflib should separate filename from timestamp with tab 107 days http://bugs.python.org/issue7585 r.david.murray patch test_cmd_line fails with non-ascii path 107 days http://bugs.python.org/issue7605 haypo patch test_xmlrpc fails with non-ascii path 107 days http://bugs.python.org/issue7606 haypo patch Test issue 98 days http://bugs.python.org/issue7621 ezio.melotti test_uuid is invalid 95 days http://bugs.python.org/issue7650 skrah patch Remove precision restriction for integer formatting. 65 days http://bugs.python.org/issue7873 mark.dickinson easy Documentation of math.pow, math.hypot, and complex.__abs__ 6 days http://bugs.python.org/issue7947 mark.dickinson patch strftime bug when timedelta is negative 45 days http://bugs.python.org/issue8026 amaury.forgeotdarc urlunparse(urlparse('x://')) now returns 'x:' instead of 'x:// 7 days http://bugs.python.org/issue8339 orsenthil easy test_tag_configure fails on FreeBSD 7 days http://bugs.python.org/issue8344 loewis buildbot Old Version Name in Interactive Mode 2 days http://bugs.python.org/issue8346 georg.brandl absolute_import future directive ignored by 2to3 0 days http://bugs.python.org/issue8358 benjamin.peterson % formating - TypeError: not all arguments converted during st 3 days http://bugs.python.org/issue8359 dpm doc: unittest.skipTest method added in 2.7 0 days http://bugs.python.org/issue8360 georg.brandl patch Remove assert in functools.total_ordering 0 days http://bugs.python.org/issue8361 ezio.melotti patch Lots of duplicate code between test_classify_oldstyle and test 0 days http://bugs.python.org/issue8363 exarkun patch, needs review Update site.setquit docstring 2 days http://bugs.python.org/issue8364 merwok patch 'readline' module fails to build on OS X - some recent change 0 days http://bugs.python.org/issue8365 l0nwlf test_winsound: failure on systems without soundcard 1 days http://bugs.python.org/issue8367 skrah patch Memory leak on multi-threaded PyObject_CallObject 0 days http://bugs.python.org/issue8368 loewis change module "builtins" to "__builtin__" in __import__ docume 3 days http://bugs.python.org/issue8370 georg.brandl patch Some locales are unsupported 1 days http://bugs.python.org/issue8374 loewis test_distutils fails when run with -j 0 days http://bugs.python.org/issue8375 tarek Errata on page:http://docs.python.org/library/stdtypes.html 0 days http://bugs.python.org/issue8377 doerwalter if __debug__: has nonobvious behaviour when evaluating .pyo wi 2 days http://bugs.python.org/issue8379 brett.cannon StringIO.write() takes any argument and converts it to a strin 1 days http://bugs.python.org/issue8382 r.david.murray pickle is unable to encode unicode surrogates 0 days http://bugs.python.org/issue8383 haypo patch _winreg remaining in test_winsound 0 days http://bugs.python.org/issue8385 brian.curtin patch test_pickle failing 0 days http://bugs.python.org/issue8386 haypo Incomprehensible import error 1 days http://bugs.python.org/issue8389 brett.cannon tarfile.open does fails with UnicodeDecodeError if parent dir 0 days http://bugs.python.org/issue8396 haypo Modules with a dot in the filename don't load 0 days http://bugs.python.org/issue8398 ezio.melotti Document os.open 0 days http://bugs.python.org/issue8399 georg.brandl patch need console/pager module 1 days http://bugs.python.org/issue8408 r.david.murray os.system() doesn't support surrogates nor bytes 0 days http://bugs.python.org/issue8412 haypo patch namedtuple vs tuple 0 days http://bugs.python.org/issue8415 rhettinger os.kill on windows 1763 days http://bugs.python.org/issue1220212 brian.curtin patch Thread shutdown exception in Thread.notify() 798 days http://bugs.python.org/issue1722344 brett.cannon patch Top Issues Most Discussed (10) ______________________________ 30 Improve GIL in 2.7 13 days open http://bugs.python.org/issue8299 23 diff.py produce unified format by default 7 days open http://bugs.python.org/issue8355 23 test_os causes delayed failure on x86 gentoo buildbot: Unknown 454 days open http://bugs.python.org/issue4970 19 Some locales are unsupported 1 days closed http://bugs.python.org/issue8374 17 RFC2732 support for urlparse (IPv6 addresses) 689 days closed http://bugs.python.org/issue2987 15 IGNORE_EXCEPTION_DETAIL should ignore the module name 124 days open http://bugs.python.org/issue7490 11 siginterrupt with flag=False is reset when signal received 7 days open http://bugs.python.org/issue8354 10 curses crash on FreeBSD 144 days open http://bugs.python.org/issue7384 9 Make some setup.py paths exclude-able 2 days open http://bugs.python.org/issue8406 9 Memory leak on multi-threaded PyObject_CallObject 0 days closed http://bugs.python.org/issue8368 From guido at python.org Fri Apr 16 18:14:22 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 09:14:22 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 8:35 AM, Stefan Behnel wrote: > Guido van Rossum, 16.04.2010 16:33: >> >> I am fine with >> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >> the ** mechanism. > > It's a bit like letting keys like 'not-an-identifier' pass through, though, > isn't it? Not really. It's hard to imagine(*) an implementation that naturally can represent strings that look like identifiers but not other strings -- typically, "identifier-ness" must be explicitly checked. But it's easy to imagine an implementation that only allows strings and not other values. ___ (*) I didn't say impossible. I don't really care about any counter-examples you may come up with -- in practice they don't matter. But the type *does* matter in practice. -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Apr 16 18:15:56 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 09:15:56 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <21741669123421670525391446873886509887-Webmail@me.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <21741669123421670525391446873886509887-Webmail@me.com> Message-ID: On Fri, Apr 16, 2010 at 8:22 AM, Ronald Oussoren wrote: > > On Friday, April 16, 2010, at 04:57PM, "Antoine Pitrou" wrote: >>Mark Dickinson gmail.com> writes: >>> >>> Okay; ?I'll open an issue for deprecation in 3.2 and removal in 3.3. >>> >>> Can this sneak in under the 'incorrect language semantics' exemption >>> for PEP 3003 (the moratorium PEP)? ?If not, then deprecation >>> presumably has to wait for 3.3. >> >>It seems that in spirit the moratorium applies more to language additions than >>to removals/limitations. The goal being that alternate implementation stop >>chasing a moving target in terms of features. >> >>So IMVHO it is fine for 3.2. > > What about 2.7, should it be deprecated there as well? I think so. Compatibility with PyPy, Jython and IronPython is only going to become more important. -- --Guido van Rossum (python.org/~guido) From brett at python.org Fri Apr 16 19:50:51 2010 From: brett at python.org (Brett Cannon) Date: Fri, 16 Apr 2010 10:50:51 -0700 Subject: [Python-Dev] patch for review: __import__ documentation Message-ID: Yes, we have different opinions. My personal take is to wait a week before you email python-dev if there has been no activity. That is enough time for people interested in the patch to get to it as we all have different schedules. Any faster and it feels like noise on the list to me. Brett (from his phone) On Apr 14, 2010 11:28 PM, "Glyph Lefkowitz" wrote: On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > I see the confusion. I think Martin meant more about open issues that required discussion, not sim... Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear to me that Martin specifically meant "simply issues that had a patch ready to go". Quoting him exactly: Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. It seems that perhaps the core developers have slightly different opinions about this? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Apr 16 20:09:37 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Apr 2010 14:09:37 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: On 4/15/2010 11:01 PM, Guido van Rossum wrote: >> pyc files inside of the `__pycache__` directories contain a magic >> identifier in their file names. These are mnemonic tags for the >> actual magic numbers used by the importer. For example, in Python >> 3.2, we could use the hexlified [10]_ magic number as a unique > > (Aside: when you search Wikipedia for "hexlify" it says "did you mean: > heavily?" :-) I regard 'hexlify', as used in binascii, to be a misspelling of 'hexify', whether it originated with Python or elsewhere. 'Hexify' itself may not be an official dictionary word but it at least follows a normal pattern of derivation. I have not bothered to say anything before because correcting the misspelling of the module function would break back compatibility. But I think its usage should stop there. Terry Jan Reedy From tjreedy at udel.edu Fri Apr 16 20:40:37 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Apr 2010 14:40:37 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1A472770E042064698CB5ADC83A12ACD39532BE6@TK5EX14MBXC118.redmond.corp.microsoft.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1A472770E042064698CB5ADC83A12ACD39532BE6@TK5EX14MBXC118.redmond.corp.microsoft.com> Message-ID: On 4/16/2010 11:22 AM, Dino Viehland wrote: > Mark Dickinson wrote: >> Removing it certainly seems in keeping with the goal of making life >> easier for alternate implementations. (Out of curiosity, does anyone >> know what IronPython does here?) >> >> I've opened http://bugs.python.org/issue8419 > > > It looks like IronPython reports a type error as well: > > IronPython 2.6.1 DEBUG (2.6.10920.0) on .NET 2.0.50727.4927 > Type "help", "copyright", "credits" or "license" for more information. >>>> def f(**kwargs): > ... print(kwargs) > ... >>>> kwargs = {1: 3} >>>> >>>> dict({}, **kwargs) > Traceback (most recent call last): > File "", line unknown, in > TypeError: expected string for dictionary argument got 1 >>>> d = {1:2} >>>> d.update({3:4}, **{5:6}) > Traceback (most recent call last): > File "", line unknown, in > TypeError: expected string for dictionary argument got 5 If the current CPython behavior is deprecated in 3.2, then I think IronPython should keep its current behavior (and future Cpython behavior) in IP 3.2 and not change it for one version. tjr From steve at holdenweb.com Fri Apr 16 20:51:30 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 14:51:30 -0400 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: <4BC8B1B2.6080808@holdenweb.com> Brian Curtin wrote: > Hi python-dev, > > The recent threads on builds/installers for Mac and Windows reminded me > of Steve Holden's push to get the python-dev team equipped via a > connection with the Microsoft Open Source Technology Center. The OSTC > team provides Microsoft Developer Network licenses to open source > projects to assist them in better supporting Windows. > >[...] > If this applies to you and you are interested, let me know. > Thanks, Brian, for taking this on. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From guido at python.org Fri Apr 16 20:52:11 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 11:52:11 -0700 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: On Fri, Apr 16, 2010 at 11:09 AM, Terry Reedy wrote: > On 4/15/2010 11:01 PM, Guido van Rossum wrote: >>> >>> pyc files inside of the `__pycache__` directories contain a magic >>> identifier in their file names. ?These are mnemonic tags for the >>> actual magic numbers used by the importer. ?For example, in Python >>> 3.2, we could use the hexlified [10]_ magic number as a unique >> >> (Aside: when you search Wikipedia for "hexlify" it says "did you mean: >> heavily?" :-) > > I regard 'hexlify', as used in binascii, to be a misspelling of 'hexify', > whether it originated with Python or elsewhere. 'Hexify' itself may not be > an official dictionary word but it at least follows a normal pattern of > derivation. I have not bothered to say anything before because correcting > the misspelling of the module function would break back compatibility. But I > think its usage should stop there. To the contrary, it was invented by Barry and ought to be added to the English language as a neologism. -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Fri Apr 16 22:43:33 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 16 Apr 2010 15:43:33 -0500 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100416120220.50cf21a9@heresy> References: <20100413162107.171c724c@heresy> <20100416120220.50cf21a9@heresy> Message-ID: 2010/4/16 Barry Warsaw : > On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >>> This feature is targeted for Python 3.2, solving the problem for those >>> and all future versions. ?It may be back-ported to Python 2.7. >> >>Is there time given that 2.7b1 was released? > > I think this would be totally up to Benjamin as the RM for 2.7. ?Although I > haven't tried yet, my sense of it is that most of the patch would port pretty > easily to trunk. ?I could probably generate a patch for review by mid-next > week. I would prefer that this not be in 2.7. The patch may be simple to port, but it represents quite a change in an eons old Python behvior. > > Whether it should or not is a different matter. ?Given that we're in beta, I'm > not sure *I* would approve it if I were the RM, but as the developer, sure, > I'd love to back port it. :) Thank you for sympathizing. :) -- Regards, Benjamin From brett at python.org Fri Apr 16 22:57:59 2010 From: brett at python.org (Brett Cannon) Date: Fri, 16 Apr 2010 13:57:59 -0700 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: I am aware my email has gone out multiple times. My phone kept saying that it was not sent, so I kept trying to force it to send. Sorry about the extra emails. On Fri, Apr 16, 2010 at 10:50, Brett Cannon wrote: > Yes, we have different opinions. My personal take is to wait a week before > you email python-dev if there has been no activity. That is enough time for > people interested in the patch to get to it as we all have different > schedules. Any faster and it feels like noise on the list to me. > > Brett (from his phone) > > On Apr 14, 2010 11:28 PM, "Glyph Lefkowitz" > wrote: > > On Apr 14, 2010, at 5:19 PM, Brett Cannon wrote: > > > I see the confusion. I think Martin meant more about open issues that > required discussion, not sim... > > Ach. I hit 'send' too soon. I also wanted to say: it seemed quite clear > to me that Martin specifically meant "simply issues that had a patch ready > to go". Quoting him exactly: > > Please understand that setting the state of an issue to "review" may *not* be the best way to trigger a review - it may be more effective to post to python-dev if you truly believe that the patch can be committed as-is. > > It seems that perhaps the core developers have slightly different opinions > about this? :) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Fri Apr 16 23:11:30 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 16 Apr 2010 14:11:30 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> Message-ID: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> >> Guido van Rossum, 16.04.2010 16:33: >>> >>> I am fine with >>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>> the ** mechanism. ISTM that making it illegal costs cycles with giving any real benefit. It is reasonably common to accept **kwds and then pass it down to another function. Do we want to validate the keys of every kwds dict on every call? Why do we even care? If I'm understanding the proposal correctly, it means that every existing application using **kwds will pay a price, either by breaking (because it uses non-string keys) or by running slower (so that every call can be checked to make sure it didn't use string keys). Raymond From benjamin at python.org Fri Apr 16 23:14:25 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 16 Apr 2010 16:14:25 -0500 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: 2010/4/16 Raymond Hettinger : > >>> Guido van Rossum, 16.04.2010 16:33: >>>> >>>> I am fine with >>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>> the ** mechanism. > > ISTM that making it illegal costs cycles with giving any real benefit. > It is reasonably common to accept **kwds and then pass it down > to another function. ?Do we want to validate the keys of every > kwds dict on every call? ?Why do we even care? > > If I'm understanding the proposal correctly, it means that > every existing application using **kwds will pay a price, either > by breaking (because it uses non-string keys) or by running > slower (so that every call can be checked to make sure it > didn't use string keys). We already pay that price for any Python function, so I'm not sure what difference adding to C, too, makes. -- Regards, Benjamin From guido at python.org Fri Apr 16 23:16:51 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 14:16:51 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 2:11 PM, Raymond Hettinger wrote: > >>> Guido van Rossum, 16.04.2010 16:33: >>>> >>>> I am fine with >>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>> the ** mechanism. > > ISTM that making it illegal costs cycles with giving any real benefit. Diasagree. The real benefit is better cross-implementation portability. > It is reasonably common to accept **kwds and then pass it down > to another function. ?Do we want to validate the keys of every > kwds dict on every call? ?Why do we even care? > > If I'm understanding the proposal correctly, it means that > every existing application using **kwds will pay a price, either > by breaking (because it uses non-string keys) or by running > slower (so that every call can be checked to make sure it > didn't use string keys). It already does this for Python functions. So there is no cost (and no change in the language semantics) except for the specific idiom involving dict, which was incorrectly taking a shortcut. -- --Guido van Rossum (python.org/~guido) From fijall at gmail.com Fri Apr 16 23:22:34 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 16 Apr 2010 15:22:34 -0600 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 3:11 PM, Raymond Hettinger wrote: > >>> Guido van Rossum, 16.04.2010 16:33: >>>> >>>> I am fine with >>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>> the ** mechanism. > > ISTM that making it illegal costs cycles with giving any real benefit. > It is reasonably common to accept **kwds and then pass it down > to another function. ?Do we want to validate the keys of every > kwds dict on every call? ?Why do we even care? > > If I'm understanding the proposal correctly, it means that > every existing application using **kwds will pay a price, either > by breaking (because it uses non-string keys) or by running > slower (so that every call can be checked to make sure it > didn't use string keys). > > > Raymond > On the other hand, we (as in alternative python implementations) are paying the price because people use it, even if only accidentally. If CPython detects such cases and complain early, it would be much easier for applications to stay cross-interpreter compatible (and I don't think it's a huge burden for them to get rid of that, django already did). Cheers, fijal From guido at python.org Fri Apr 16 23:31:25 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 14:31:25 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 2:22 PM, Maciej Fijalkowski wrote: > On Fri, Apr 16, 2010 at 3:11 PM, Raymond Hettinger > wrote: >> >>>> Guido van Rossum, 16.04.2010 16:33: >>>>> >>>>> I am fine with >>>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>>> the ** mechanism. >> >> ISTM that making it illegal costs cycles with giving any real benefit. >> It is reasonably common to accept **kwds and then pass it down >> to another function. ?Do we want to validate the keys of every >> kwds dict on every call? ?Why do we even care? >> >> If I'm understanding the proposal correctly, it means that >> every existing application using **kwds will pay a price, either >> by breaking (because it uses non-string keys) or by running >> slower (so that every call can be checked to make sure it >> didn't use string keys). >> >> >> Raymond >> > > On the other hand, we (as in alternative python implementations) are > paying the price because people use it, even if only accidentally. If > CPython detects such cases and complain early, it would be much easier > for applications to stay cross-interpreter compatible (and I don't > think it's a huge burden for them to get rid of that, django already > did). +1. Apparently dict(x, **y) is going around as "cool hack" for "call x.update(y) and return x". Personally I find it more despicable than cool. -- --Guido van Rossum (python.org/~guido) From daniel at stutzbachenterprises.com Fri Apr 16 23:42:17 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 16 Apr 2010 16:42:17 -0500 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 4:11 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > ISTM that making it illegal costs cycles with giving any real benefit. > It is reasonably common to accept **kwds and then pass it down > to another function. Do we want to validate the keys of every > kwds dict on every call? Why do we even care? > IIRC, there's a performance hack in dictobject.c that keeps track of whether all of the keys are strings or not. The hack is designed so that lookup operations can call the string compare/hash functions directly if possible, rather than going through the slower PyObject_ functions. Consequently, validating **kwds should be cheap. I don't know if the the current validating of **kwds with Python functions already leverages that hack or not. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Apr 16 23:51:30 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 16 Apr 2010 16:51:30 -0500 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: 2010/4/16 Daniel Stutzbach : > On Fri, Apr 16, 2010 at 4:11 PM, Raymond Hettinger > wrote: >> >> ISTM that making it illegal costs cycles with giving any real benefit. >> It is reasonably common to accept **kwds and then pass it down >> to another function. ?Do we want to validate the keys of every >> kwds dict on every call? ?Why do we even care? > > IIRC, there's a performance hack in dictobject.c that keeps track of whether > all of the keys are strings or not.? The hack is designed so that lookup > operations can call the string compare/hash functions directly if possible, > rather than going through the slower PyObject_ functions. > > Consequently, validating **kwds should be cheap. That won't work. You could put non-string keys in a dictionary and remove them, but the dictionary would still be in the less optimized state. -- Regards, Benjamin From raymond.hettinger at gmail.com Fri Apr 16 23:56:09 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 16 Apr 2010 14:56:09 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <3F5E8832-7237-4233-BD90-C84C2EC0BD94@gmail.com> > On Fri, Apr 16, 2010 at 2:11 PM, Raymond Hettinger > wrote: >> >>>> Guido van Rossum, 16.04.2010 16:33: >>>>> >>>>> I am fine with >>>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>>> the ** mechanism. >> >> ISTM that making it illegal costs cycles with giving any real benefit. > > Diasagree. The real benefit is better cross-implementation portability. Would hate for 100% of users will pay a performance penalty when most applications aren't abusing keyword dictionaries so they already work cross-platfrom. Isn't there anyway to make this a one-time check instead of a PyLint style validation check running on every invocation of a function or method using **kwds? Or perhaps there can be a switch or flag to enable developers to check for non-standard uses of **kwds. That way, they can clean-up their programs but not make the end users pay for the checks every time they run an application that has already been cleaned-up. Raymond From daniel at stutzbachenterprises.com Fri Apr 16 23:57:19 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Fri, 16 Apr 2010 16:57:19 -0500 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 4:51 PM, Benjamin Peterson wrote: > 2010/4/16 Daniel Stutzbach : > > IIRC, there's a performance hack in dictobject.c that keeps track of > whether > > all of the keys are strings or not. The hack is designed so that lookup > > operations can call the string compare/hash functions directly if > possible, > > rather than going through the slower PyObject_ functions. > > That won't work. You could put non-string keys in a dictionary and > remove them, but the dictionary would still be in the less optimized > state. > It would be enough to make the common case fast (1 pointer comparison). The error case would need to check all of the keys, true, but that's not really a performance concern. Unless you're saying you often create a dictionary, add non-string keys, remove the non-string keys, then pass it as a **kwds? ;-) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Apr 17 00:02:58 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 16 Apr 2010 18:02:58 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <20100416220307.B80493A4063@sparrow.telecommunity.com> At 04:51 PM 4/16/2010 -0500, Benjamin Peterson wrote: >That won't work. You could put non-string keys in a dictionary and >remove them, but the dictionary would still be in the less optimized state. That only means it's slower on uncommon cases and the case where you're about to get an exception anyway. From solipsis at pitrou.net Sat Apr 17 00:09:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 16 Apr 2010 22:09:44 +0000 (UTC) Subject: [Python-Dev] Very Strange Argument Handling Behavior References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <3F5E8832-7237-4233-BD90-C84C2EC0BD94@gmail.com> Message-ID: Raymond Hettinger gmail.com> writes: > > Would hate for 100% of users will pay a performance penalty > when most applications aren't abusing keyword dictionaries > so they already work cross-platfrom. Someone should provide actual measurements before we start a psychodrama about Python performance being brutally butchered and chopped into small pieces on the altar of inter-implementation compatibility ;) Regards Antoine. From raymond.hettinger at gmail.com Sat Apr 17 00:40:48 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 16 Apr 2010 15:40:48 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Apr 16, 2010, at 2:42 PM, Daniel Stutzbach wrote: > > IIRC, there's a performance hack in dictobject.c that keeps track of whether all of the keys are strings or not. The hack is designed so that lookup operations can call the string compare/hash functions directly if possible, rather than going through the slower PyObject_ functions. > > Consequently, validating **kwds should be cheap. > Good thinking. That would definitely be better than scanning the full dict on every call. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Apr 17 01:00:30 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 16 Apr 2010 19:00:30 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <20100416190030.074c429b@heresy> On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >> Byte code files contain two 32-bit numbers followed by the marshaled > >big-endian Done. >> [2]_ code object. ?The 32-bit numbers represent a magic number and a >> timestamp. ?The magic number changes whenever Python changes the byte >> code format, e.g. by adding new byte codes to its virtual machine. >> This ensures that pyc files built for previous versions of the VM >> won't cause problems. ?The timestamp is used to make sure that the pyc >> file is not older than the py file that was used to create it. ?When > >is not older than -> matches > >(Obscure fact: the timestamp in the pyc file must match the source's >mtime exactly.) Done. >> Rationale >> ========= >> >> Linux distributions such as Ubuntu [4]_ and Debian [5]_ provide more >> than one Python version at the same time to their users. ?For example, >> Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, >> with Python 2.6 being the default. >> >> This causes a conflict for Python source files installed by the >> system (including third party packages), because you cannot compile a > >I'd say only 3rd part packages right? (And code written by the distro, >which from Python's POV is also 3rd party.) At least ought to clarify >that the stdlib is unaffected by this conflict, because multiple >versions of the stdlib *are* installed. Yes, good point. Clarified. >> single Python source file for more than one Python version at a time. >> Thus if your system wanted to install a `/usr/share/python/foo.py`, it >> could not create a `/usr/share/python/foo.pyc` file usable across all >> installed Python versions. > >Note that (due to the magic#) Python doesn't crash, it just falls back >on the slower approach of compiling from source. > >Perhaps more important is that different Python versions (if the user >has write permission) will fight over the pyc file and rewrite it each >time the source is compiled. Worse, even though the magic# is >initially written as zero and then rewritten with the correct value, >concurrent processes running different Python versions can actually >end up reading corrupt bytecode. (Alex Martelli diagnosed this at >Google years ago.) Good point; I've made this more clear. >> Furthermore, in order to ease the burden on operating system packagers >> for these distributions, the distribution packages do not contain >> Python version numbers [6]_; they are shared across all Python >> versions installed on the system. ?Putting Python version numbers in >> the packages would be a maintenance nightmare, since all the packages >> - *and their dependencies* - would have to be updated every time a new >> Python release was added or removed from the distribution. ?Because of >> the sheer number of packages available, this amount of work is >> infeasible. >> >> C extensions can be source compatible across multiple versions of >> Python. ?Compiled extension modules are usually not compatible though, > >Actually we typically make every effort to support backwards >compatibility for compiled modules, and the module initialization API >contains a version# check. This is a different version# than the >import magic# and historically has changed much less frequently. I've rewritten this paragraph a bit. It's not particularly relevant to this PEP. (I'll be look at PEP 384 soon.) >> and PEP 384 [7]_ has been proposed to address this by defining a >> stable ABI for extension modules. >> >> Proposal >> ======== >> >> Python's import machinery is extended to write and search for byte >> code cache files in a single directory inside every Python package >> directory. ?This directory will be called `__pycache__`. >> Further, pyc files will contain a magic string that differentiates the > >Clarify that the magic string is in the filename, not in the file contents. Yep. >> Python version they were compiled for. ?This allows multiple byte >> compiled cache files to co-exist for a single Python source file. >> >> This scheme has the added benefit of reducing the clutter in a Python >> package directory. >> >> When a Python source file is imported for the first time, a >> `__pycache__` directory will be created in the package directory, if > >Is this still true? ISTR there was a lot of discussion about the >auto-creation and possible security concerns. It is still true. I think we determined it will usually not be an issue because the umask will not be altered, and because normal installation procedures typically involve byte compilation (and thus __pycache__ creation) during installation time via tools like compileall. This really is describing what happens when you run Python over pure Python source code for the first time, and it's no different from what happens now with the automatic creation of pyc files. >> one does not already exist. ?The pyc file for the imported source will >> be written to the `__pycache__` directory, using the magic-tag > >By now the magic-tag format should have been defined (or a "see below" >inserted). Based on this and your following comment, I've moved the description of the magic tag format to here, and rewritten it to fit in context. The section discussing the hexadecimal representation is moved to the (rejected) "Alternatives" section. >> Case 1: The first import >> ------------------------ >> >> When Python is asked to import module `foo`, it searches for a >> `foo.py` file (or `foo` package, but that's not important for this >> discussion) along its `sys.path`. ?When Python locates the `foo.py` >> file it will look for a `__pycache__` directory in the directory where >> it found the `foo.py`. ?If the `__pycache__` directory is missing, >> Python will create it. ?Then it will parse and byte compile the >> `foo.py` file and save the byte code in `__pycache__/foo..pyc`, >> where is defined by the Python implementation, but will be a >> human readable string such as `cpython-32`. > >(Aside: at first I read this as a description of the full algorithm. >But there is a step missing -- the __pycache__/foo..pyc file is >searched and not found.) I added a Case 0 for the "steady state" which should clarify this. >> Magic identifiers >> ================= >> >> pyc files inside of the `__pycache__` directories contain a magic >> identifier in their file names. ?These are mnemonic tags for the >> actual magic numbers used by the importer. ?For example, in Python >> 3.2, we could use the hexlified [10]_ magic number as a unique > >(Aside: when you search Wikipedia for "hexlify" it says "did you mean: >heavily?" :-) :) Emacs is where I first encountered this term, e.g. M-x hexlify-buffer. It got carried over to the binascii module. But in this case "hexadecimal representation of the binary magic number" is probably a better term to use. > >> identifier:: >> >> ? ?>>> from binascii import hexlify >> ? ?>>> from imp import get_magic >> ? ?>>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii')) >> ? ?'foo.580c0d0a.pyc' >> >> This isn't particularly human friendly though. ?Instead, this PEP > >This section reads a bit weird -- first it describes the solution we >*didn't* pick. I'd move that to a "Alternatives Considered and >Rejected" section or some such. Agreed; see above. >> proposes a *magic tag* that uniquely defines `.pyc` files for the >> current version of Python. ?Whenever the magic number is bumped, a new >> magic tag is defined which is unique among all versions and >> implementations of Python. ?The actual contents of the magic tag is >> left up to the implementation, although it is recommended that the tag >> include the implementation name and a version shorthand. ?In general, >> magic numbers never change between Python micro releases, but the >> convention can be extended to handle magic number changes between >> pre-release development versions. >> >> For example, CPython 3.2 would have a magic tag of `cpython-32` and >> write pyc files like this: `foo.cpython-32.pyc`. ?When the `-O` flag >> is used, it would write `foo.cpython-32.pyo`. ?For backports of this >> feature to Python 2, when the `-U` flag is used, a file such as >> `foo.cpython-27u.pyc` can be written. > >Does all of this match the implementation? Yes. Well, except for the -U part, since I haven't backported this to Python 2... yet :). >> Implementation strategy >> ======================= >> >> This feature is targeted for Python 3.2, solving the problem for those >> and all future versions. ?It may be back-ported to Python 2.7. > >Is there time given that 2.7b1 was released? See my previous response. >> This PEP proposes the addition of an `__cached__` attribute to >> modules, which will always point to the actual `pyc` file that was >> read or written. ?When the environment variable >> `$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if >> the source lives on a read-only filesystem, then the `__cached__` >> attribute will point to the location that the `pyc` file *would* have >> been written to if it didn't exist. ?This location of course includes >> the `__pycache__` subdirectory in its path. > >Hm. I wish there was a way to find out whether the bytecode (or >whatever) actually *was* read from this file. __file__ in Python 2 >supports this (though not in Python 3). Do you have a use case for that? It might be interesting to know, but I can't think of a good way to infer that from __file__ and __cached__, or of a good way to expose that on module objects. Of course, it would be totally Python implementation dependent too. >> Backports >> --------- >> >> For versions of Python earlier than 3.2 (and possibly 2.7), it is >> possible to backport this PEP. ?However, in Python 3.2 (and possibly >> 2.7), this behavior will be turned on by default, and in fact, it will >> replace the old behavior. ?Backports will need to support the old >> layout by default. ?We suggest supporting PEP 3147 through the use of >> an environment variable called `$PYTHONENABLECACHEDIR` or the command >> line switch `-Xenablecachedir` to enable the feature. > >I would be okay if a distro decided to turn it on by default, as long >as there was a way to opt out. For Python 2.6, even for a distro-specific backport, I think I'd want to enable it only with a switch. It might be weird for example if Python 2.6 in Ubuntu 10.04 produced traditional pyc files, but Python 2.6 in Ubuntu 10.10 produced PEP 3147 file names. For a backport to Python 2.7 though (which e.g. would be new in Ubuntu 10.10), it might make sense to enable it by default. Either way, we're really talking about the effects on user code only. We'll definitely enable it as part of the package installation tools. Thanks again Guido. I think this hits all your feedback. Now to land the code! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Sat Apr 17 01:07:07 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 16 Apr 2010 19:07:07 -0400 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: References: <20100413162107.171c724c@heresy> Message-ID: <20100416190707.65d694d5@heresy> On Apr 16, 2010, at 11:52 AM, Guido van Rossum wrote: >To the contrary, it was invented by Barry and ought to be added to the >English language as a neologism. Actually, it's an Emacs invention! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From steve at holdenweb.com Sat Apr 17 01:38:44 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 16 Apr 2010 19:38:44 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: Raymond Hettinger wrote: > > On Apr 16, 2010, at 2:42 PM, Daniel Stutzbach wrote: >> >> IIRC, there's a performance hack in dictobject.c that keeps track of >> whether all of the keys are strings or not. The hack is designed so >> that lookup operations can call the string compare/hash functions >> directly if possible, rather than going through the slower PyObject_ >> functions. >> >> Consequently, validating **kwds should be cheap. >> > Good thinking. > > That would definitely be better than scanning the full dict on every call. > I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) >>> def f(**kwargs): ... kwargs[1] = "dummy" ... print(kwargs) ... >>> f(this="Guido", that="Raymond", the_other="Steve") {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} >>> Or would we? If it's OK to mutate kwargs inside the function to contain a non-string key, why isn't it OK to pass a non-string key in? I understand that it couldn't be generated using keyword argument syntax, but I don't see why we discriminate against f(**dict(...)) to limit it to what could be generated using keyword argument syntax. Is this such a big deal? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Sat Apr 17 01:42:07 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 17 Apr 2010 01:42:07 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <4BC8F5CF.1050502@voidspace.org.uk> On 17/04/2010 01:38, Steve Holden wrote: > Raymond Hettinger wrote: > >> On Apr 16, 2010, at 2:42 PM, Daniel Stutzbach wrote: >> >>> IIRC, there's a performance hack in dictobject.c that keeps track of >>> whether all of the keys are strings or not. The hack is designed so >>> that lookup operations can call the string compare/hash functions >>> directly if possible, rather than going through the slower PyObject_ >>> functions. >>> >>> Consequently, validating **kwds should be cheap. >>> >>> >> Good thinking. >> >> That would definitely be better than scanning the full dict on every call. >> >> > I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) > > >>>> def f(**kwargs): >>>> > ... kwargs[1] = "dummy" > ... print(kwargs) > ... > >>>> f(this="Guido", that="Raymond", the_other="Steve") >>>> > {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} > >>>> > Or would we? Nobody is proposing barring that. > If it's OK to mutate kwargs inside the function to contain > a non-string key, why isn't it OK to pass a non-string key in? > > Because they are completely different operations. Michael > I understand that it couldn't be generated using keyword argument > syntax, but I don't see why we discriminate against f(**dict(...)) to > limit it to what could be generated using keyword argument syntax. Is > this such a big deal? > > regards > Steve > -- http://www.ironpythoninaction.com/ From victor.stinner at haypocalc.com Sat Apr 17 02:12:50 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 17 Apr 2010 02:12:50 +0200 Subject: [Python-Dev] Drop OS/2 support? Message-ID: <201004170212.50400.victor.stinner@haypocalc.com> Hi, Python contains code specific to OS/2 (eg. see Modules/posixmodule.c). I read in Wikipedia that IBM has discontinued OS/2 support in 2005. Do we still support OS/2 or not? I'm asking because I'm working on a patch modifying OS2 specific code, but I'm unable to compile nor test my changes. And on IRC we told me that nobody knows/uses OS/2. If we support OS/2, we need a buildbot. My patch: http://bugs.python.org/issue8391 (os.execvpe() doesn't support surrogates in env). -- Victor Stinner http://www.haypocalc.com/ From greg.ewing at canterbury.ac.nz Sat Apr 17 02:43:13 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 17 Apr 2010 12:43:13 +1200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <4BC90421.80704@canterbury.ac.nz> Daniel Stutzbach wrote: > Unless you're saying you often create a dictionary, add non-string keys, > remove the non-string keys, then pass it as a **kwds? ;-) I think the point is that it would create a very mysterious potential failure mode. What would you make of a situation where Python says "TypeError: Keyword dict contains non-string keys", but upon examination, the dict clearly does not contain any such thing? -- Greg From skip at pobox.com Sat Apr 17 02:36:15 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 16 Apr 2010 19:36:15 -0500 Subject: [Python-Dev] Drop OS/2 support? In-Reply-To: <201004170212.50400.victor.stinner@haypocalc.com> References: <201004170212.50400.victor.stinner@haypocalc.com> Message-ID: <19401.639.66684.323294@montanaro.dyndns.org> Victor> Do we still support OS/2 or not? Yes. Andrew MacIntyre maintains the OS/2 port: http://www.andymac.org/ He's on python-dev. You mostly see activity from him around release time. I would contact him directly if you need some assistance from him. Skip From fuzzyman at voidspace.org.uk Sat Apr 17 02:53:58 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 17 Apr 2010 02:53:58 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <4BC90421.80704@canterbury.ac.nz> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC90421.80704@canterbury.ac.nz> Message-ID: <4BC906A6.4060008@voidspace.org.uk> On 17/04/2010 02:43, Greg Ewing wrote: > Daniel Stutzbach wrote: > >> Unless you're saying you often create a dictionary, add non-string >> keys, remove the non-string keys, then pass it as a **kwds? ;-) > > I think the point is that it would create a very mysterious > potential failure mode. What would you make of a situation > where Python says "TypeError: Keyword dict contains non-string > keys", but upon examination, the dict clearly does not contain > any such thing? > No, if the dictionary is not marked as an all-string dict it can fallback to checking. The common case (dict marked as all strings) is fast. Michael -- http://www.ironpythoninaction.com/ From guido at python.org Sat Apr 17 04:44:55 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 19:44:55 -0700 Subject: [Python-Dev] PEP 3147 ready for pronouncement and merging In-Reply-To: <20100416190030.074c429b@heresy> References: <20100413162107.171c724c@heresy> <20100416190030.074c429b@heresy> Message-ID: Thanks for all the changes! On Fri, Apr 16, 2010 at 4:00 PM, Barry Warsaw wrote: > On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >>Hm. I wish there was a way to find out whether the bytecode (or >>whatever) actually *was* read from this file. __file__ in Python 2 >>supports this (though not in Python 3). > > Do you have a use case for that? ?It might be interesting to know, but I can't > think of a good way to infer that from __file__ and __cached__, or of a good > way to expose that on module objects. ? Of course, it would be totally Python > implementation dependent too. The only use case I can think of is a unit test that would indirectly assess that bytecode was (or wasn't) read in specific conditions. You can safely ignore this use case. -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Apr 17 04:47:05 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Apr 2010 19:47:05 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <3F5E8832-7237-4233-BD90-C84C2EC0BD94@gmail.com> References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <3F5E8832-7237-4233-BD90-C84C2EC0BD94@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 2:56 PM, Raymond Hettinger wrote: > >> On Fri, Apr 16, 2010 at 2:11 PM, Raymond Hettinger >> wrote: >>> >>>>> Guido van Rossum, 16.04.2010 16:33: >>>>>> >>>>>> I am fine with >>>>>> declaring dict({}, **{1:3}) illegal, since after all it is abuse of >>>>>> the ** mechanism. >>> >>> ISTM that making it illegal costs cycles with giving any real benefit. >> >> Diasagree. The real benefit is better cross-implementation portability. > > Would hate for 100% of users will pay a performance penalty > when most applications aren't abusing keyword dictionaries > so they already work cross-platfrom. > > Isn't there anyway to make this a one-time check instead > of a PyLint style validation check running on every invocation > of a function or method using **kwds? > > Or perhaps there can be a switch or flag to enable developers to > check for non-standard uses of **kwds. ?That way, they can clean-up > their programs but not make the end users pay for the checks every time > they run an application that has already been cleaned-up. Please stop worrying. -- --Guido van Rossum (python.org/~guido) From andymac at bullseye.apana.org.au Sat Apr 17 05:19:36 2010 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sat, 17 Apr 2010 13:19:36 +1000 Subject: [Python-Dev] Drop OS/2 support? In-Reply-To: <201004170212.50400.victor.stinner@haypocalc.com> References: <201004170212.50400.victor.stinner@haypocalc.com> Message-ID: <4BC928C8.7060007@bullseye.andymac.org> Victor Stinner wrote: > Python contains code specific to OS/2 (eg. see Modules/posixmodule.c). I read > in Wikipedia that IBM has discontinued OS/2 support in 2005. Do we still > support OS/2 or not? As was recently discussed, what constitutes "support" varies in perception. Python's source contains support for it to be built on OS/2, but python.org's normal build process doesn't deliver binaries. OS/2 lives on as eCS. It is a second tier platform, akin to Cygwin (which also has some source level support but is not part of the normal python.org release process). > I'm asking because I'm working on a patch modifying OS2 specific code, but I'm > unable to compile nor test my changes. And on IRC we told me that nobody > knows/uses OS/2. > > If we support OS/2, we need a buildbot. Build-bots for first tier platforms (Windows, Linux & OS/X) are useful for the python.org developer community. IMO build-bots for secondary platforms are the more the responsibility of that platform's community on the basis of best management of python.org resources. I don't think anyone reasonable would expect patch suppliers to deal with second tier platforms - I certainly don't. It is nice to get heads-up messages about issues that might involve such support though, and it shouldn't take much searching to find me to enquire. > My patch: http://bugs.python.org/issue8391 (os.execvpe() doesn't support > surrogates in env). I'll look at it when I get a chance, but I'm not expecting that my input should affect your patch and I'm not expecting you to try and deal with the issue on OS/2. The 3.x branch needs quite a bit of work on OS/2 to deal with Unicode, as OS/2 was one of the earlier OSes with full multiple language support and IBM developed a unique API. I'm still struggling to come to terms with this, partly because I myself don't "need" it. Regards, Andrew. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From ncoghlan at gmail.com Sat Apr 17 11:33:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Apr 2010 19:33:33 +1000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <4BC9806D.7040302@gmail.com> Steve Holden wrote: > I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) > >>>> def f(**kwargs): > ... kwargs[1] = "dummy" > ... print(kwargs) > ... >>>> f(this="Guido", that="Raymond", the_other="Steve") > {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} > > Or would we? If it's OK to mutate kwargs inside the function to contain > a non-string key, why isn't it OK to pass a non-string key in? Because we can easily enough add the kwargs type check to CPython to improve cross-implementation compatibility, while we can't easily check individual assignments. > I understand that it couldn't be generated using keyword argument > syntax, but I don't see why we discriminate against f(**dict(...)) to > limit it to what could be generated using keyword argument syntax. Is > this such a big deal? The language spec is intended to allow the use of string-only dictionaries for namespaces. CPython happens to use normal dictionaries and then optimises them heavily when they only contain string keys, but other implementations are meant to be allowed to use a separate type that doesn't permit arbitrary keys. In cases where it is easy for CPython to check for purely string keys, that's probably worth doing (especially now it has been noted that the common case of that check passing can be made fast using the existing string keys only optimisation tools). There's also a consistency argument here, in that CPython already performs this check for Python functions, so it is anomalous that there are ways to do this without triggering the TypeError. More exotic namespace abuse is generally harder to detect, and almost certainly not worth the hassle of flagging on CPython itself (such code will still fail on implementations that actually use string-key only namespaces). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Apr 17 11:39:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Apr 2010 19:39:27 +1000 Subject: [Python-Dev] Drop OS/2 support? In-Reply-To: <201004170212.50400.victor.stinner@haypocalc.com> References: <201004170212.50400.victor.stinner@haypocalc.com> Message-ID: <4BC981CF.2010307@gmail.com> Victor Stinner wrote: > If we support OS/2, we need a buildbot. As Andrew said, there are 2 levels of support - the "if we break it, we'll fix it" level where our buildbots live, and the "we won't go out of our way to break it, but it may degenerate as other things change (or simply miss out on some new features)" level. Eventually there's the "trying not to break this is actively causing problems, so now we're breaking it on purpose" level of explicitly not supporting a platform. Relatively recent examples of that were stripping out the Win9x support, as well as the culling of old code specific to various proprietary Unix implementions. I don't know how well this is articulated in the relevant PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Sat Apr 17 11:42:07 2010 From: eric at trueblade.com (Eric Smith) Date: Sat, 17 Apr 2010 05:42:07 -0400 Subject: [Python-Dev] Drop OS/2 support? In-Reply-To: <4BC928C8.7060007@bullseye.andymac.org> References: <201004170212.50400.victor.stinner@haypocalc.com> <4BC928C8.7060007@bullseye.andymac.org> Message-ID: <4BC9826F.9080704@trueblade.com> Andrew MacIntyre wrote: > It is nice to get > heads-up messages about issues that might involve such support though, > and it shouldn't take much searching to find me to enquire. Especially since aimacintyre is listed in Misc/maintainers.rst. From python-dev at masklinn.net Sat Apr 17 12:53:34 2010 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 17 Apr 2010 12:53:34 +0200 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <966F26F8-E3EA-405D-8DB7-F1E86A2F5BB9@masklinn.net> On 16 Apr 2010, at 23:31 , Guido van Rossum wrote: > > +1. > > Apparently dict(x, **y) is going around as "cool hack" for "call > x.update(y) and return x". Personally I find it more despicable than > cool. This description doesn't make sense since `dict(x, **y)` returns not an updated `x` but a new dictionary merging `x` and `y`. And that's how (and why) I use it, it's simpler (and ? I believe ? more readable) to write `z = dict(x, **y)` than `z = dict(x); z.update(y)`, since Python doesn't overload addition as it does for lists: l3 = l1 + l2 works and is equivalent to l3 = list(l1); l3.extend(l2) but there is no easy way to say that with dicts, at the moment. From vinay_sajip at yahoo.co.uk Sat Apr 17 14:41:03 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 17 Apr 2010 12:41:03 +0000 (UTC) Subject: [Python-Dev] Very Strange Argument Handling Behavior References: <4BC81A16.6050107@zhuliguan.net> <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: Steve Holden holdenweb.com> writes: > I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) > > >>> def f(**kwargs): > ... kwargs[1] = "dummy" > ... print(kwargs) > ... > >>> f(this="Guido", that="Raymond", the_other="Steve") > {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} I think that according to the proposal, the above snippet would be OK, but def f(**kwargs): kwargs[1] = 'dummy' g(**kwargs) would fail at the call of g. Regards, Vinay Sajip From guido at python.org Sat Apr 17 17:22:19 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Apr 2010 08:22:19 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Fri, Apr 16, 2010 at 4:38 PM, Steve Holden wrote: > I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) > >>>> def f(**kwargs): > ... ? kwargs[1] = "dummy" > ... ? print(kwargs) > ... >>>> f(this="Guido", that="Raymond", the_other="Steve") > {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} >>>> > > Or would we? If it's OK to mutate kwargs inside the function to contain > a non-string key, why isn't it OK to pass a non-string key in? Because Python promises that the object the callee sees as 'kwargs' is "just a dict". But the requirement for what the caller passes in is different: it must pass in a dict whose keys represent argument names. If you want an API where you can pass in an arbitrary dict to be received unchanged, don't use **kw. Remember that the caller can independently decide whether or not to use the **kw notation -- there is no implied correspondence between the caller's use of **kw and the callee's use of it. Note this example: def f(a, b, **k): print(a, b, k) d = {'a': 1, 'b': 2, 'c': 3} f(**d) This will print 1 2 {'c': 3} Note that the k received by f is not the same as the d passed in! (And yet d of course is not modified by the operation.) > I understand that it couldn't be generated using keyword argument > syntax, but I don't see why we discriminate against f(**dict(...)) to > limit it to what could be generated using keyword argument syntax. Is > this such a big deal? Is portability of code to Jython, IronPython, PyPy a big deal? According to a slide you recently posted to the PSF board list, it is. -- --Guido van Rossum (python.org/~guido) From steve at holdenweb.com Sat Apr 17 17:24:42 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 17 Apr 2010 11:24:42 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <4BC9D2BA.5040900@holdenweb.com> Guido van Rossum wrote: > On Fri, Apr 16, 2010 at 4:38 PM, Steve Holden wrote: >> I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) >> >>>>> def f(**kwargs): >> ... kwargs[1] = "dummy" >> ... print(kwargs) >> ... >>>>> f(this="Guido", that="Raymond", the_other="Steve") >> {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} >> Or would we? If it's OK to mutate kwargs inside the function to contain >> a non-string key, why isn't it OK to pass a non-string key in? > > Because Python promises that the object the callee sees as 'kwargs' is > "just a dict". But the requirement for what the caller passes in is > different: it must pass in a dict whose keys represent argument names. > > If you want an API where you can pass in an arbitrary dict to be > received unchanged, don't use **kw. Remember that the caller can > independently decide whether or not to use the **kw notation -- there > is no implied correspondence between the caller's use of **kw and the > callee's use of it. Note this example: > > def f(a, b, **k): > print(a, b, k) > > d = {'a': 1, 'b': 2, 'c': 3} > f(**d) > > This will print > > 1 2 {'c': 3} > > Note that the k received by f is not the same as the d passed in! (And > yet d of course is not modified by the operation.) > Good point, and one I hadn't thought of. I was blithely assuming that the dict seen by the called function was always what was passed as the dict argument. >> I understand that it couldn't be generated using keyword argument >> syntax, but I don't see why we discriminate against f(**dict(...)) to >> limit it to what could be generated using keyword argument syntax. Is >> this such a big deal? > > Is portability of code to Jython, IronPython, PyPy a big deal? > According to a slide you recently posted to the PSF board list, it is. > And I haven't changed my mind since. Thanks for the comprehensive response. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From guido at python.org Sat Apr 17 17:27:18 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Apr 2010 08:27:18 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: On Sat, Apr 17, 2010 at 5:41 AM, Vinay Sajip wrote: > Steve Holden holdenweb.com> writes: > >> I'm sure we wouldn't want to go so far as to inhibit this. (Py 3.1) >> >> >>> def f(**kwargs): >> ... ? kwargs[1] = "dummy" >> ... ? print(kwargs) >> ... >> >>> f(this="Guido", that="Raymond", the_other="Steve") >> {'this': 'Guido', 1: 'dummy', 'the_other': 'Steve', 'that': 'Raymond'} > > I think that according to the proposal, the above snippet would be OK, but > > def f(**kwargs): > ? ?kwargs[1] = 'dummy' > ? ?g(**kwargs) > > would fail at the call of g. And that is already the status quo. Try it out in your friendly Python interpreter. The *only* thing that should be *changed* is for the dict() builtin to insist that if it receives a dict containing keyword arguments the keys must all be strings. This *only* affects the dict() builtin. It has keyword arguments so that instead of {'foo': 1, 'bar': 2} you can write dict(foo=1, bar=2), which arguably is more readable because it doesn't have so many quotes. OTOH calling dict(x, **y) is a weird hack. Note that if you wrote a wrapper function in Python around dict() with the same behavior there would be no way to prevent the check that all the keys are strings. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Apr 17 18:22:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Apr 2010 02:22:54 +1000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BC8711F.8060001@gmail.com> <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> Message-ID: <4BC9E05E.1070609@gmail.com> Guido van Rossum wrote: > Because Python promises that the object the callee sees as 'kwargs' is > "just a dict". Huh, I thought kwargs was allowed to be implemented as a string-keys-only dict (similar to class and module namespaces) while still be a valid Python implementation. I guess I was wrong. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From guido at python.org Sat Apr 17 18:49:25 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Apr 2010 09:49:25 -0700 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <4BC9E05E.1070609@gmail.com> References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> Message-ID: On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan wrote: > Guido van Rossum wrote: >> Because Python promises that the object the callee sees as 'kwargs' is >> "just a dict". > > Huh, I thought kwargs was allowed to be implemented as a > string-keys-only dict (similar to class and module namespaces) while > still be a valid Python implementation. I guess I was wrong. Actually I don't know about that. Is there language anywhere in the language reference that says this? What do IronPython, Jython, PyPy actually do? In any case my line of reasoning in this case isn't affected by this; as I pointed out in my reply to Steve, the relation between the **k passed in by the caller and the **k received by the callee is less direct than you might think. The proposed change *only* affects the dict() builtin; any change in the type of **k seen by the callee would potentially affect all user-defined functions. -- --Guido van Rossum (python.org/~guido) From victor.stinner at haypocalc.com Sat Apr 17 19:17:29 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 17 Apr 2010 19:17:29 +0200 Subject: [Python-Dev] http://bugs.python.org/ is down Message-ID: <201004171917.29252.victor.stinner@haypocalc.com> Hi, http://bugs.python.org/ displays "Service Temporarily Unavailable". Is it normal? -- Victor Stinner http://www.haypocalc.com/ From daniel at stutzbachenterprises.com Sat Apr 17 19:19:51 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sat, 17 Apr 2010 12:19:51 -0500 Subject: [Python-Dev] http://bugs.python.org/ is down In-Reply-To: <201004171917.29252.victor.stinner@haypocalc.com> References: <201004171917.29252.victor.stinner@haypocalc.com> Message-ID: On Sat, Apr 17, 2010 at 12:17 PM, Victor Stinner < victor.stinner at haypocalc.com> wrote: > http://bugs.python.org/ displays "Service Temporarily Unavailable". Is it > normal? > It's working fine for me. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Apr 17 19:33:12 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 17 Apr 2010 12:33:12 -0500 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> Message-ID: 2010/4/17 Guido van Rossum : > On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan wrote: >> Guido van Rossum wrote: >>> Because Python promises that the object the callee sees as 'kwargs' is >>> "just a dict". >> >> Huh, I thought kwargs was allowed to be implemented as a >> string-keys-only dict (similar to class and module namespaces) while >> still be a valid Python implementation. I guess I was wrong. > > Actually I don't know about that. Is there language anywhere in the > language reference that says this? What do IronPython, Jython, PyPy > actually do? Similar to CPython, PyPy has dict versions optimized for strings, which fall back to the general version when given non-string keys. -- Regards, Benjamin From dinov at microsoft.com Sat Apr 17 19:38:14 2010 From: dinov at microsoft.com (Dino Viehland) Date: Sat, 17 Apr 2010 17:38:14 +0000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> Message-ID: <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> Benjamin wrote: > 2010/4/17 Guido van Rossum : > > On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan > wrote: > >> Guido van Rossum wrote: > >>> Because Python promises that the object the callee sees as 'kwargs' > is > >>> "just a dict". > >> > >> Huh, I thought kwargs was allowed to be implemented as a > >> string-keys-only dict (similar to class and module namespaces) while > >> still be a valid Python implementation. I guess I was wrong. > > > > Actually I don't know about that. Is there language anywhere in the > > language reference that says this? What do IronPython, Jython, PyPy > > actually do? > > Similar to CPython, PyPy has dict versions optimized for strings, > which fall back to the general version when given non-string keys. IronPython as well. The only place we use a string only dict is for new-style classes whose dict's are wrapped in a dictproxy. From fijall at gmail.com Sat Apr 17 19:52:52 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 17 Apr 2010 11:52:52 -0600 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> Message-ID: On Sat, Apr 17, 2010 at 11:38 AM, Dino Viehland wrote: > Benjamin wrote: >> 2010/4/17 Guido van Rossum : >> > On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan >> wrote: >> >> Guido van Rossum wrote: >> >>> Because Python promises that the object the callee sees as 'kwargs' >> is >> >>> "just a dict". >> >> >> >> Huh, I thought kwargs was allowed to be implemented as a >> >> string-keys-only dict (similar to class and module namespaces) while >> >> still be a valid Python implementation. I guess I was wrong. >> > >> > Actually I don't know about that. Is there language anywhere in the >> > language reference that says this? What do IronPython, Jython, PyPy >> > actually do? >> >> Similar to CPython, PyPy has dict versions optimized for strings, >> which fall back to the general version when given non-string keys. > > IronPython as well. ?The only place we use a string only dict is for > new-style classes whose dict's are wrapped in a dictproxy. And yet that breaks some code :-) From techtonik at gmail.com Sat Apr 17 20:02:44 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 17 Apr 2010 21:02:44 +0300 Subject: [Python-Dev] OS information, tags In-Reply-To: <4BC7E405.9080304@holdenweb.com> References: <20100412135151.GA31685@remy> <4BC46996.4020909@gmail.com> <4BC7E405.9080304@holdenweb.com> Message-ID: On Fri, Apr 16, 2010 at 7:13 AM, Steve Holden wrote: > Brian Curtin wrote: >> On Thu, Apr 15, 2010 at 03:20, anatoly techtonik > > wrote: >> >> ? ? On Tue, Apr 13, 2010 at 3:54 PM, Nick Coghlan > ? ? > wrote: >> ? ? > >> ? ? >>> ? ? ? ? ? ?I am surprised to see that the bug-tracker >> ? ? >>> doesn't have an OS classifier or ability to add >> ? ? >>> tags ? Since a number of issues reported seem to >> >> ? ? Just to remind about my +1 for user editable tags. >> >> >> -sys.maxint on just about anything user editable except for the title or >> messages on an issue. I don't think user editable tags on an issue bring >> anything to the table except making it seem more "web 2.0". Searching >> user-generated tags would be a nightmare because you'd have to know >> every combination of some idea in order to get relevant results (e.g., >> windows, windoze, window$, win32). >> > > Yes, tight vocabulary control will lead to more consistent classifications. There can be an interface to normalize tags. Known synonyms can be merged to the primary list automatically warning user about substitution. If word is not in primary list the user may propose to add it to main list. Tags with most user proposals will clearly indicate candidates for inclusion. Users still should be able to tag issues with primary set of tags or propose a tag for an issue. This will partially take triaging burden off the core developers and will help to spot people with most proposals as potential contributors who deserve more tracker privileges. We need tags gameplay to get people addicted. -- anatoly t. From dinov at microsoft.com Sat Apr 17 20:03:59 2010 From: dinov at microsoft.com (Dino Viehland) Date: Sat, 17 Apr 2010 18:03:59 +0000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> Message-ID: <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> Maciej wrote: > On Sat, Apr 17, 2010 at 11:38 AM, Dino Viehland > wrote: > > Benjamin wrote: > >> 2010/4/17 Guido van Rossum : > >> > On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan > >> wrote: > >> >> Guido van Rossum wrote: > >> >>> Because Python promises that the object the callee sees as > 'kwargs' > >> is > >> >>> "just a dict". > >> >> > >> >> Huh, I thought kwargs was allowed to be implemented as a > >> >> string-keys-only dict (similar to class and module namespaces) > while > >> >> still be a valid Python implementation. I guess I was wrong. > >> > > >> > Actually I don't know about that. Is there language anywhere in > the > >> > language reference that says this? What do IronPython, Jython, > PyPy > >> > actually do? > >> > >> Similar to CPython, PyPy has dict versions optimized for strings, > >> which fall back to the general version when given non-string keys. > > > > IronPython as well. ?The only place we use a string only dict is for > > new-style classes whose dict's are wrapped in a dictproxy. > > And yet that breaks some code :-) Sure, if you do: class C(object): locals()[object()] = 42 dir(C) You lose. Once I'm aware of some piece of code in the wild doing this then I'll be happy to change IronPython to be more compatible. :) From fijall at gmail.com Sat Apr 17 20:10:12 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 17 Apr 2010 12:10:12 -0600 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> References: <4BC9E05E.1070609@gmail.com> <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> Message-ID: On Sat, Apr 17, 2010 at 12:03 PM, Dino Viehland wrote: > Maciej wrote: >> On Sat, Apr 17, 2010 at 11:38 AM, Dino Viehland >> wrote: >> > Benjamin wrote: >> >> 2010/4/17 Guido van Rossum : >> >> > On Sat, Apr 17, 2010 at 9:22 AM, Nick Coghlan >> >> wrote: >> >> >> Guido van Rossum wrote: >> >> >>> Because Python promises that the object the callee sees as >> 'kwargs' >> >> is >> >> >>> "just a dict". >> >> >> >> >> >> Huh, I thought kwargs was allowed to be implemented as a >> >> >> string-keys-only dict (similar to class and module namespaces) >> while >> >> >> still be a valid Python implementation. I guess I was wrong. >> >> > >> >> > Actually I don't know about that. Is there language anywhere in >> the >> >> > language reference that says this? What do IronPython, Jython, >> PyPy >> >> > actually do? >> >> >> >> Similar to CPython, PyPy has dict versions optimized for strings, >> >> which fall back to the general version when given non-string keys. >> > >> > IronPython as well. ?The only place we use a string only dict is for >> > new-style classes whose dict's are wrapped in a dictproxy. >> >> And yet that breaks some code :-) > > Sure, if you do: > > class C(object): > ? ?locals()[object()] = 42 > > dir(C) > > You lose. ?Once I'm aware of some piece of code in the wild doing this > then I'll be happy to change IronPython to be more compatible. :) > There was one thing in sqlalchemy tests, not sure exactly why. There were also other things that I've seen, but consequently it was decided that it's only accidentally working on CPython and namespace should contain string-only keys. Cheers, fijal From bluemangroupie at gmail.com Sat Apr 17 20:17:39 2010 From: bluemangroupie at gmail.com (aditya bhargava) Date: Sat, 17 Apr 2010 13:17:39 -0500 Subject: [Python-Dev] http://bugs.python.org/ is down In-Reply-To: References: <201004171917.29252.victor.stinner@haypocalc.com> Message-ID: Works for me too. On Sat, Apr 17, 2010 at 12:19 PM, Daniel Stutzbach < daniel at stutzbachenterprises.com> wrote: > On Sat, Apr 17, 2010 at 12:17 PM, Victor Stinner < > victor.stinner at haypocalc.com> wrote: > >> http://bugs.python.org/ displays "Service Temporarily Unavailable". Is it >> normal? >> > > It's working fine for me. > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/bluemangroupie%40gmail.com > > -- wefoundland.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Sat Apr 17 20:41:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 17 Apr 2010 20:41:10 +0200 Subject: [Python-Dev] bbreport Message-ID: <201004172041.10219.victor.stinner@haypocalc.com> Hi, Ezio and Florent are developing a tool called bbreport to collect buildbot results and generate short reports to the command line. It's possible to filter results by Python branch, builder name, etc. I send patches to link failed tests to existing issues to see quickly known failures vs new failures. This tool becomes really useful to analyze buildbot results! http://code.google.com/p/bbreport/ bbreport requires Python trunk (2.7) and color output only works on UNIX/BSD OS (ie. not Windows). -- Victor Stinner http://www.haypocalc.com/ From steve at holdenweb.com Sat Apr 17 21:03:44 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 17 Apr 2010 15:03:44 -0400 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> References: <1BEFF1C0-73C4-4CEF-A06C-DA94515B52FA@gmail.com> <4BC9E05E.1070609@gmail.com> <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> Message-ID: <4BCA0610.2070104@holdenweb.com> Dino Viehland wrote: > Maciej wrote: [...] >> And yet that breaks some code :-) > > Sure, if you do: > > class C(object): > locals()[object()] = 42 > > dir(C) > > You lose. Once I'm aware of some piece of code in the wild doing this > then I'll be happy to change IronPython to be more compatible. :) > > This would be a lose anyway, since the CPython specifications suggest that you should not rely on being able to change locals() (or at least shouldn't expect that such changes are actually reflected in the local namespace). regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From dickinsm at gmail.com Sat Apr 17 22:04:00 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 17 Apr 2010 21:04:00 +0100 Subject: [Python-Dev] bbreport In-Reply-To: <201004172041.10219.victor.stinner@haypocalc.com> References: <201004172041.10219.victor.stinner@haypocalc.com> Message-ID: On Sat, Apr 17, 2010 at 7:41 PM, Victor Stinner wrote: > Ezio and Florent are developing a tool called bbreport to collect buildbot > results and generate short reports to the command line. It's possible to > filter results by Python branch, builder name, etc. I send patches to link > failed tests to existing issues to see quickly known failures vs new failures. > This tool becomes really useful to analyze buildbot results! Seconded. I've been using this for a few days, and found it especially useful to be able to get a quick summary of exactly *which* tests are failing on the various buildbots. > bbreport requires Python trunk (2.7) and color output only works on UNIX/BSD > OS (ie. not Windows). Does it really need trunk? I've been running it under 2.6 without problems, but I probably haven't explored all the options fully. Mark From db3l.net at gmail.com Sat Apr 17 22:19:51 2010 From: db3l.net at gmail.com (David Bolen) Date: Sat, 17 Apr 2010 16:19:51 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> <4BC7F70A.9000503@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > This actually happened on Windows - some people now > recommend to run the buildbot scripts on a regular developer checkout, > because they supposedly do the right things. I have to admit that I'm guilty of this (though to be fair most of my test builds are buildbot-related), if only because it takes care of all the external stuff automatically and self-contained in the build tree. > I would rather prefer to have the buildbot run the same commands that we > recommend developers to run. The knowledge about these should be in the > README, and it should be stable knowledge, i.e. require infrequent > changes. This is true for Unix: configure/make/make test/make clean > had been the correct procedure for ten years now. The Unix buildbot only > deviates slightly, to have the slaves run a more expensive version of > the test suite. In digging around a bit, it would appear that there's actually quite a bit of OSX support already in the Makefile (either the main one or the one in Mac). There's even a target that tests both halves of a universal build (using rosetta for the PPC version) on an Intel box. I suspect it's just a question of setting up a Mac-appropriate builder, using the universal SDK in the configure step, and whatever Makefile targets are deemed best and current, perhaps with the addition of support for pulling in the third party stuff through externals or whatever. A first pass could just be to factor that process into a separate stage of build-installer that could be run independently of the rest of the installer build process. In the interim, I've generated the third-party libraries via the current trunk build-installer script and installed them in /usr/local on my buildbot, so they are found by any of the buildbot builds using the stock configure/make approach. Other than a slight update to the bzip version, looks like the dependency versions in the installer script haven't changed for over a year, so I suspect the libraries are fine for any of the branches currently being built. I also updated to the latest 8.4.19 Tcl/Tk in /Library/Frameworks since I saw some interpreter crashes in tests in what appeared to be a Tcl code path. It had been building against my system 8.4.7 Tcl. The Windows buildbot uses Tcl 8.5 - not sure if that should be preferred for the Mac buildbot as well, but will leave it at 8.4 for now. I think this should create test builds more representative of the final binaries. It's not testing a universal framework build, but the test target only tests the Intel path anyway, the generated code should still be the same, and the same libraries are in use. If anyone more familiar with this side of things for OSX has some spare time down the road, I'd be happy to help work on improving the process for the buildbot. > I'd be interested in setting up daily builds then. For these, I think > it's fine that they may differ slightly from the official binaries - > people would recognize that they are testing a different set of binaries. We can certainly start by just trying to automate the execution of build-installer, something I suspect can be completely controlled from the master side, and then uploading the resulting dmg file. I'd be happy to help coordinate any experiments offline if you're interested. I do currently have a DMG built for 2.7 Beta 1, if it would be useful. -- David From steve at holdenweb.com Sat Apr 17 22:29:50 2010 From: steve at holdenweb.com (Steve Holden) Date: Sat, 17 Apr 2010 16:29:50 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> <4BC7F70A.9000503@v.loewis.de> Message-ID: David Bolen wrote: > "Martin v. L?wis" writes: > >> This actually happened on Windows - some people now >> recommend to run the buildbot scripts on a regular developer checkout, >> because they supposedly do the right things. > > I have to admit that I'm guilty of this (though to be fair most of my > test builds are buildbot-related), if only because it takes care of > all the external stuff automatically and self-contained in the build > tree. > >> I would rather prefer to have the buildbot run the same commands that we >> recommend developers to run. The knowledge about these should be in the >> README, and it should be stable knowledge, i.e. require infrequent >> changes. This is true for Unix: configure/make/make test/make clean >> had been the correct procedure for ten years now. The Unix buildbot only >> deviates slightly, to have the slaves run a more expensive version of >> the test suite. > > In digging around a bit, it would appear that there's actually quite a > bit of OSX support already in the Makefile (either the main one or > the one in Mac). There's even a target that tests both halves of a > universal build (using rosetta for the PPC version) on an Intel box. > > I suspect it's just a question of setting up a Mac-appropriate > builder, using the universal SDK in the configure step, and whatever > Makefile targets are deemed best and current, perhaps with the > addition of support for pulling in the third party stuff through > externals or whatever. A first pass could just be to factor that > process into a separate stage of build-installer that could be run > independently of the rest of the installer build process. > > In the interim, I've generated the third-party libraries via the > current trunk build-installer script and installed them in /usr/local > on my buildbot, so they are found by any of the buildbot builds using > the stock configure/make approach. Other than a slight update to the > bzip version, looks like the dependency versions in the installer > script haven't changed for over a year, so I suspect the libraries are > fine for any of the branches currently being built. > > I also updated to the latest 8.4.19 Tcl/Tk in /Library/Frameworks > since I saw some interpreter crashes in tests in what appeared to be a > Tcl code path. It had been building against my system 8.4.7 Tcl. The > Windows buildbot uses Tcl 8.5 - not sure if that should be preferred > for the Mac buildbot as well, but will leave it at 8.4 for now. > > I think this should create test builds more representative of the > final binaries. It's not testing a universal framework build, but the > test target only tests the Intel path anyway, the generated code > should still be the same, and the same libraries are in use. > > If anyone more familiar with this side of things for OSX has some > spare time down the road, I'd be happy to help work on improving the > process for the buildbot. > >> I'd be interested in setting up daily builds then. For these, I think >> it's fine that they may differ slightly from the official binaries - >> people would recognize that they are testing a different set of binaries. > > We can certainly start by just trying to automate the execution of > build-installer, something I suspect can be completely controlled from > the master side, and then uploading the resulting dmg file. I'd be > happy to help coordinate any experiments offline if you're interested. > > I do currently have a DMG built for 2.7 Beta 1, if it would be useful. > Great work David, this is a terrific step forward. Thanks! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fijall at gmail.com Sat Apr 17 23:47:08 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 17 Apr 2010 15:47:08 -0600 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: <4BCA0610.2070104@holdenweb.com> References: <4BC9E05E.1070609@gmail.com> <1A472770E042064698CB5ADC83A12ACD3953825C@TK5EX14MBXC118.redmond.corp.microsoft.com> <1A472770E042064698CB5ADC83A12ACD39538470@TK5EX14MBXC118.redmond.corp.microsoft.com> <4BCA0610.2070104@holdenweb.com> Message-ID: On Sat, Apr 17, 2010 at 1:03 PM, Steve Holden wrote: > Dino Viehland wrote: >> Maciej wrote: > [...] >>> And yet that breaks some code :-) >> >> Sure, if you do: >> >> class C(object): >> ? ? locals()[object()] = 42 >> >> dir(C) >> >> You lose. ?Once I'm aware of some piece of code in the wild doing this >> then I'll be happy to change IronPython to be more compatible. :) >> >> > This would be a lose anyway, since the CPython specifications suggest > that you should not rely on being able to change locals() (or at least > shouldn't expect that such changes are actually reflected in the local > namespace). You can override __new__ of a type subclass to achieve the same effect (or even directly call type.__new__ with strange dict as an argument). Cheers, fijal From steve at pearwood.info Sun Apr 18 00:21:57 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 18 Apr 2010 08:21:57 +1000 Subject: [Python-Dev] Very Strange Argument Handling Behavior In-Reply-To: References: <4BCA0610.2070104@holdenweb.com> Message-ID: <201004180821.58911.steve@pearwood.info> On Sun, 18 Apr 2010 07:47:08 am Maciej Fijalkowski wrote: > On Sat, Apr 17, 2010 at 1:03 PM, Steve Holden wrote: > > Dino Viehland wrote: > >> Maciej wrote: > > > > [...] > > > >>> And yet that breaks some code :-) > >> > >> Sure, if you do: > >> > >> class C(object): > >> ? ? locals()[object()] = 42 > >> > >> dir(C) > >> > >> You lose. ?Once I'm aware of some piece of code in the wild doing > >> this then I'll be happy to change IronPython to be more > >> compatible. :) > > > > This would be a lose anyway, since the CPython specifications > > suggest that you should not rely on being able to change locals() > > (or at least shouldn't expect that such changes are actually > > reflected in the local namespace). > > You can override __new__ of a type subclass to achieve the same > effect (or even directly call type.__new__ with strange dict as an > argument). I think that only works in Python 3.x, in 2.x the __dict__ is always a regular dict no matter what you pass. At least for CPython. If there is a way to set the dict of a class to something other than a regular dict in CPython 2.x, I would be very pleased! -- Steven D'Aprano From Tobias.Herp at gmx.de Sun Apr 18 12:31:02 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Sun, 18 Apr 2010 12:31:02 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action Message-ID: <4BCADF66.8070702@gmx.de> Hi, all, (I don't know whether the first attempt to post this went through; I could'nt find it, so I retry. My apologies for any inconvenience...) I noticed that Python 2.7 beta 1 now contains the argparse module, which might be a good thing. The code has been cleaned up, too, compared to the current version 1.1. But there is still one issue with argparse; to tell the story from the beginning: The ArgumentParser class uses uses '-v' as a default short option for the 'tell me the program version' facility. Since this is commonly used for verbosity, and the vast majority of *X commandline tools use '-V' instead (if they bother to accompany '--version' with a short option at all). Therefore I submitted an issue report at . To put it short: *Argparse should simply do this like optparse does already.* I.e., use '--version', '--help' and '-h' by default, but not '-v'. Subsequently I spent a hard time arguing. (Perhaps it was somewhat distracting that I proposed a way of postponing the creation of the help and version options and to provide them with the short options which are left unused e.g. for hosts (database clients), human readability (ls, du, etc.) or verbosity; Steven had stated it might break backward compatibility if the '-v' option string for version output would be dropped which might be relied on by other programs which parse the output. IMO, however, anyone should be beaten who relies on '-v' giving the version while '--version' has been present all the time. The Zen of Python, 2nd aphorism: "Explicit is better than implicit") What happened was the following: Completely unnecessarily, the 'version' constructor argument is now deprecated. This fact doesn't solve any problem I can think of; the only effect is to make programming more cumbersome, and it is /one more/ difference to optparse. The 'version' argument is a perfectly reasonable way to provide a script with a simple version information feature. Of course, it should only define the '--version' argument; it *must not* define '-v' for this purpose, since this is commonly used for verbosity, and '-V' is widely used to accompany '--version'. I have lots of scripts which use optparse, and every single one uses the version argument. I consider this a matter of good style. The deprecation of the 'version' argument press-gangs people to replace it by parser.add_argument('--version', action='version', version='', # the only common part help="show program's version number and exit") in every single script (which violates the DRY principle, by the way). Of course, if a more fancy version information is needed, e.g. reporting the versions of all imported modules, it is perfectly possible to just omit the version argument during creation and build a special 'version' action. No deprecation warnings are needed for this. *Before Python 2.7 reaches productivity stage*, IMNSHO the following changes should be applied to argparse.py: - removal of the deprecation warnings - removal of the default usage of '-v' with the version information facility This is a very simple thing to do; I'd happily provide a patch. (The only complicated task might be to make the unit tests reflect the change; but for a start, 7 lines of the test_main function could be dropped.) Just for the records, this is what optparse does: - it defines -h and --help for the help (unless suppressed) - it defines --version for the version (if version argument given) This is how it should be (regarding the version, at least). This is how the vast majority of *x tools looks like. No reason to change this behaviour. What do you think? -- Cheers, Tobias From ncoghlan at gmail.com Sun Apr 18 13:03:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Apr 2010 21:03:34 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCADF66.8070702@gmx.de> References: <4BCADF66.8070702@gmx.de> Message-ID: <4BCAE706.2090503@gmail.com> Tobias Herp wrote: > *Before Python 2.7 reaches productivity stage*, IMNSHO the following > changes should be applied to argparse.py: > > - removal of the deprecation warnings > - removal of the default usage of '-v' > with the version information facility > > This is a very simple thing to do; I'd happily provide a patch. > (The only complicated task might be to make the unit tests reflect the > change; but for a start, 7 lines of the test_main function could be > dropped.) > > Just for the records, this is what optparse does: > - it defines -h and --help for the help (unless suppressed) > - it defines --version for the version (if version argument given) > This is how it should be (regarding the version, at least). > This is how the vast majority of *x tools looks like. > No reason to change this behaviour. > > What do you think? I can see Steven's argument from an argparse point of view, but I think he is taking backwards compatibility to an unhealthy extreme. Deprecating a perfectly valid option *just* because someone might be relying on -v instead of -V as the shorthand for version? A phrase involving babies and bathwater comes to mind... I would be a lot happier if argparse as a standard library module just followed optparse's lead here (i.e. defined '--version' only and punted on the shorthand form). To deal with this in a backwards compatible way while remaining on the path to more conventional behaviour, I suggest the following: 1. For Python 2.7, deprecate *just* the "-v" default behaviour for the version. This means "--version" and "-v" will be set to invoke different actions when the version argument is supplied (the latter will trigger a deprecation warning if supplied, while the former will work normally). 2. For Python 3.2, don't create the "-v" shorthand at all (i.e. only create "--version"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From george.sakkis at gmail.com Sun Apr 18 15:03:53 2010 From: george.sakkis at gmail.com (George Sakkis) Date: Sun, 18 Apr 2010 15:03:53 +0200 Subject: [Python-Dev] Keyword-only arguments in 2.x Message-ID: Hi all, what's the status of backporting PEP 3102 ? It was supposed to go into 2.6 and it seems it won't be in 2.7 either. There is an updated patch in the tracker [1] that applies cleanly on the latest trunk and passes all relevant tests, so unless there has been a decision against backporting it, would it be possible to push it for 2.7 ? I think this is a very useful PEP and it would be great to have it in 2.x too. George [1] http://bugs.python.org/issue1745 From ben+python at benfinney.id.au Sun Apr 18 15:16:55 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 18 Apr 2010 23:16:55 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action References: <4BCADF66.8070702@gmx.de> <4BCAE706.2090503@gmail.com> Message-ID: <87ochgj3y0.fsf@benfinney.id.au> Nick Coghlan writes: > I would be a lot happier if argparse as a standard library module just > followed optparse's lead here (i.e. defined '--version' only and punted > on the shorthand form). > > To deal with this in a backwards compatible way while remaining on the > path to more conventional behaviour, I suggest the following: > > 1. For Python 2.7, deprecate *just* the "-v" default behaviour for the > version. This means "--version" and "-v" will be set to invoke different > actions when the version argument is supplied (the latter will trigger a > deprecation warning if supplied, while the former will work normally). > > 2. For Python 3.2, don't create the "-v" shorthand at all (i.e. only > create "--version"). +1 As someone who uses the existing ?optparse? behaviour to implement a number of Unix command-line programs, the above all makes more sense for ?argparse? in the standard library if we're expecting it to be a smooth upgrade. -- \ ?Natural catastrophes are rare, but they come often enough. We | `\ need not force the hand of nature.? ?Carl Sagan, _Cosmos_, 1980 | _o__) | Ben Finney From Tobias.Herp at gmx.de Sun Apr 18 11:37:39 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Sun, 18 Apr 2010 11:37:39 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action Message-ID: <4BCAD2E3.90102@gmx.de> Hi, all, I noticed that Python 2.7 beta 1 now contains the argparse module, which might be a good thing. The code has been cleaned up, too, compared to the current version 1.1. But there is still one issue with argparse; to tell the story from the beginning: The ArgumentParser class uses uses '-v' as a default short option for the 'tell me the program version' facility. Since this is commonly used for verbosity, and the vast majority of *X commandline tools use '-V' instead (if they bother to accompany '--version' with a short option at all). Therefore I submitted an issue report at . To put it short: *Argparse should simply do this like optparse does already.* I.e., use '--version', '--help' and '-h' by default, but not '-v'. Subsequently I spent a hard time arguing. (Perhaps it was somewhat distracting that I proposed a way of postponing the creation of the help and version options and to provide them with the short options which are left unused e.g. for hosts (database clients), human readability (ls, du, etc.) or verbosity; Steven had stated it might break backward compatibility if the '-v' option string for version output would be dropped which might be relied on by other programs which parse the output. IMO, however, anyone should be beaten who relies on '-v' giving the version while '--version' has been present all the time. The Zen of Python, 2nd aphorism: "Explicit is better than implicit") What happened was the following: Completely unnecessarily, the 'version' constructor argument is now deprecated. This fact doesn't solve any problem I can think of; the only effect is to make programming more cumbersome, and it is /one more/ difference to optparse. The 'version' argument is a perfectly reasonable way to provide a script with a simple version information feature. Of course, it should only define the '--version' argument; it *must not* define '-v' for this purpose, since this is commonly used for verbosity, and '-V' is widely used to accompany '--version'. I have lots of scripts which use optparse, and every single one uses the version argument. I consider this a matter of good style. The deprecation of the 'version' argument press-gangs people to replace it by parser.add_argument('--version', action='version', version='', # the only common part help="show program's version number and exit") in every single script (which violates the DRY principle, by the way). Of course, if a more fancy version information is needed, e.g. reporting the versions of all imported modules, it is perfectly possible to just omit the version argument during creation and build a special 'version' action. No deprecation warnings are needed for this. *Before Python 2.7 reaches productivity stage*, IMNSHO the following changes should be applied to argparse.py: - removal of the deprecation warnings - removal of the default usage of '-v' with the version information facility This is a very simple thing to do; I'd happily provide a patch. (The only complicated task might be to make the unit tests reflect the change; but for a start, 7 lines of the test_main function could be dropped.) Just for the records, this is what optparse does: - it defines -h and --help for the help (unless suppressed) - it defines --version for the version (if version argument given) This is how it should be (regarding the version, at least). This is how the vast majority of *x tools looks like. No reason to change this behaviour. What do you think? -- Cheers, Tobias From Tobias.Herp at gmx.de Sun Apr 18 15:51:00 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Sun, 18 Apr 2010 15:51:00 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <87ochgj3y0.fsf@benfinney.id.au> References: <4BCADF66.8070702@gmx.de> <4BCAE706.2090503@gmail.com> <87ochgj3y0.fsf@benfinney.id.au> Message-ID: <4BCB0E44.6080901@gmx.de> Ben Finney schrieb: > Nick Coghlan writes: > >> I would be a lot happier if argparse as a standard library module just >> followed optparse's lead here (i.e. defined '--version' only and punted >> on the shorthand form). >> >> To deal with this in a backwards compatible way while remaining on the >> path to more conventional behaviour, I suggest the following: >> >> 1. For Python 2.7, deprecate *just* the "-v" default behaviour for the >> version. This means "--version" and "-v" will be set to invoke different >> actions when the version argument is supplied (the latter will trigger a >> deprecation warning if supplied, while the former will work normally). >> >> 2. For Python 3.2, don't create the "-v" shorthand at all (i.e. only >> create "--version"). > > +1 > > As someone who uses the existing ?optparse? behaviour to implement a > number of Unix command-line programs, the above all makes more sense for > ?argparse? in the standard library if we're expecting it to be a smooth > upgrade. Guys, thanks for your feedback. Let me see; the suggestion involves the '-v' being set in Python 2.7 by default, but to another action, which yields a DeprecationWarning /and/ performs the version action, right? The drawback would be: The '-v' wouldn't be available for other uses. But I use it all the time (for verbosity). I don't see why we shouldn't drop the '-v' altogether; where Python standard libraries are concerned, argparse is a new module, and no compatibility can be broken; people will expect this to follow the optparse habits. Programs developed against former versions of argparse will continue to work, too; there will just not be a (non-standard) short option generated, but the long option '--version' will be still there. Honestly, how frequently do you query program versions? Unfortunately adding a new '-V' default action /could/ break existing programs, since people might have worked around the former default; thus we couldn't do this. Let's get rid of the default '-v' /now/. This is clean, this is easy, it won't break anything (with the possible exception of "prog -v" version output parsers who deserve no better), and we would be done with it. -- Tobias From ronaldoussoren at mac.com Sun Apr 18 16:13:39 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 18 Apr 2010 16:13:39 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC63599.5020005@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> Message-ID: On 14 Apr, 2010, at 23:37, Michael Foord wrote: > On 14/04/2010 23:32, Greg Ewing wrote: >> Michael Foord wrote: >>> Building Python requires, I believe, the XCode development tools to be installed. Even then, building a full version of Python - with *all* the C extensions that are part of a Python release - is not a trivial task. >> >> What's non-trivial about it? I usually find that the normal >> "./configure; make; make install" sequence works fine without >> any further intervention. >> >> If you want a framework installation you have to read the >> README and use a couple of extra options, but it still works >> very smoothly. >> > A build on my machine produces output similar to: > > > Python build finished, but the necessary bits to build these modules were not found: > _bsddb dl gdbm > imageop linuxaudiodev ossaudiodev > readline spwd sunaudiodev > To find the necessary bits, look in setup.py in detect_modules() for the module's name. > > > Failed to build these modules: > _tkinter > > Obviously many of those are not meant to be built and I usually build Python for running the test suite - so I don't care about not having Tkinter. A new user of Python would most certainly care about not having Tkinter. What's the OS version? Do you have a copy of Tcl/Tk in /Library/Frameworks? Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From ronaldoussoren at mac.com Sun Apr 18 16:22:41 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 18 Apr 2010 16:22:41 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <9C0AE3C0-D860-4501-8B4C-47C4614580A9@zope.com> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC607AB.4080203@v.loewis.de> <4BC617C5.7010907@voidspace.org.uk> <4BC61973.9020207@v.loewis.de> <35AEF1A0-5202-4925-8184-3C418596C788@zope.com> <4BC62833.4010609@v.loewis.de> <9C0AE3C0-D860-4501-8B4C-47C4614580A9@zope.com> Message-ID: On 15 Apr, 2010, at 0:12, Zvezdan Petkovic wrote: > On Apr 14, 2010, at 4:40 PM, Martin v. L?wis wrote: > >>> I think you just need to supply to configure >>> >>> MACOSX_DEPLOYMENT_TARGET=10.4 >>> >>> and have the appropriate SDK installed with Xcode. >> >> Wouldn't that break 10.3 compatibility (seel below)? > > > I was replying to your point about 10.4 build. > Naturally, if you want a 10.3 build you'd pass 10.3 as the target and would have to have appropriate Xcode SDK installed. You don't have to install an SDK to be able to build binaries that run on older versions. The reason the binary installer gets build with the 10.4u SDK is that the default compiler on OSX 10.4 cannot build universal binaries without that SDK. > > >>>> Unfortunately, Apple manages to break compatibility and portability >>>> with every release, which makes this particular build task soooo >>>> tricky. You have to make all kinds of decisions and compromises >>>> where are really difficult to keep track of. >>> >>> >>> Hmm. Apple provided compatibility SDK and documented it. >>> >>> The only compromise I see in this build process right now is that we >>> are building a Panther (10.3) compatible installer, while Mac OS X is >>> a certified UNIX starting with 10.5. >> >> I think there are more issues. People want a fat binary that supports >> AMD64 along with x86, yet building such a binary requires an SDK that >> won't support PPC anymore - right? > > Yes. No. It is possible to build a binary that supports ppc, ppc64, x86 and x86_64, and that's even possible using a single additional configure switch. That binary will require OSX 10.5 to run though due to using symbols that aren't available on earlier versions of OSX (thanks to the better UNIX API compatibility in 10.5). PPC64 is not supported on OSX 10.6 though. > > x86_64, i386, and ppc are supported even in the Xcode supplied with the latest Mac OS X 10.6. Only ppc64 is not. So, ppc is not an issue. > > The problem is that enforcing backward compatibility with 10.3 and 10.4 makes 64-bit Intel architecture not feasible. > > You are right, it is a compromise. > We are making more users happy by providing a 32-bit installer for a wider range of OS releases. > > However, if we want a more modern certified UNIX, 64-bit installer, then we'll have to draw a line and stop supporting older OS releases. > > Just as we stop supporting older releases of Python. I want to provide 2 installers for Python 2.7 and 3.2: 1) The current 32-bit only installer that runs on OSX 10.3 or later 2) An installer that supports ppc, x86 and x86_64 and requires OSX 10.5 or later The latter would be the one that most users would want to use. Note that the second installer does not support ppc64 and three reasons: (1) PPC64 is a dead end on OSX, (2) libffi has issues on darwin/ppc64 that probably affect ctypes and (3) I do not have regular access to ppc64 machines and can therefore not provide any support for that platform. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From ronaldoussoren at mac.com Sun Apr 18 16:25:19 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 18 Apr 2010 16:25:19 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BC697D2.4020200@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> Message-ID: On 15 Apr, 2010, at 6:36, Martin v. L?wis wrote: > Greg Ewing wrote: >> Michael Foord wrote: >>> Building Python requires, I believe, the XCode development tools to be >>> installed. Even then, building a full version of Python - with *all* >>> the C extensions that are part of a Python release - is not a trivial >>> task. >> >> What's non-trivial about it? > > Building a DMG file, in a way that the output will actually work on most > other systems. That *is* trivial: use Mac/BuildScript/build-installer.py on OSX 10.5. Making sure that unrelated changes don't accidently break the OSX build can be non-trivial though... That doesn't work with the py3k trunk at the moment, I just noticed that framework builds are broken there. I've filed issue #8441 for that and am working on a fix. Ronald > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From techtonik at gmail.com Sun Apr 18 16:56:30 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 18 Apr 2010 17:56:30 +0300 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: Twisted folks will surely appreciate any help and may be able to contribute back. http://twistedmatrix.com/trac/wiki/Windows -- anatoly t. On Fri, Apr 16, 2010 at 4:01 PM, Brian Curtin wrote: > Hi python-dev, > > The recent threads on builds/installers for Mac and Windows reminded me of > Steve Holden's push to get the python-dev team equipped via a connection > with the Microsoft Open Source Technology Center. The OSTC team provides > Microsoft Developer Network licenses to open source projects to assist them > in better supporting Windows. > > I've talked with Steve (who passed the task to me) and the Microsoft folks, > and they are happy to provide more licenses if needed. If you are interested > in getting a copy of MSDN, please contact me off-list. I'll provide you with > a code that you'll put into their site, then around a week later you should > get your subscription. > > The snippet below is taken from prior correspondence with the OSTC team in > regards to who can receive the licenses: > > """ > For the purposes of providing MSDN licenses to an open source development > community, I consider anyone who writes, builds, tests or documents software > to be a "developer who contributes" to the project. (In fact, having started > out as a test engineer, I would take exception to anyone who claimed only > people who write code are "developers" :-) We do ask that requests are for > people who are active contributors and not just minor/occasional > participants. > """ > > If this applies to you and you are interested, let me know. > > Brian Curtin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/techtonik%40gmail.com > > From martin at v.loewis.de Sun Apr 18 17:15:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 Apr 2010 17:15:29 +0200 Subject: [Python-Dev] Keyword-only arguments in 2.x In-Reply-To: References: Message-ID: <4BCB2211.9040405@v.loewis.de> > what's the status of backporting PEP 3102 ? It looks like it won't happen. > It was supposed to go into > 2.6 and it seems it won't be in 2.7 either. Correct - and therefore, it won't be backported at all. > There is an updated patch > in the tracker [1] that applies cleanly on the latest trunk and passes > all relevant tests, so unless there has been a decision against > backporting it, would it be possible to push it for 2.7 ? No - 2.7 has already seen its first beta release, so no new features are acceptable (except by release manager decision or BDFL pronouncement). Regards, Martin From exarkun at twistedmatrix.com Sun Apr 18 17:16:52 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 18 Apr 2010 15:16:52 -0000 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: <20100418151652.2682.989910686.divmod.xquotient.4@localhost.localdomain> On 02:56 pm, techtonik at gmail.com wrote: >Twisted folks will surely appreciate any help and may be able to >contribute back. >http://twistedmatrix.com/trac/wiki/Windows Extra Windows and VS licenses would certainly be helpful for Twisted development, and might lead indirectly to CPython/Windows improvements. Is this the kind of use for which it is appropriate to request an MSDN license via the PSF? Jean-Paul From martin at v.loewis.de Sun Apr 18 17:17:31 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 Apr 2010 17:17:31 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> Message-ID: <4BCB228B.2080904@v.loewis.de> Ronald Oussoren wrote: > On 15 Apr, 2010, at 6:36, Martin v. L?wis wrote: > >> Greg Ewing wrote: >>> Michael Foord wrote: >>>> Building Python requires, I believe, the XCode development tools to be >>>> installed. Even then, building a full version of Python - with *all* >>>> the C extensions that are part of a Python release - is not a trivial >>>> task. >>> What's non-trivial about it? >> Building a DMG file, in a way that the output will actually work on most >> other systems. > > That *is* trivial: use Mac/BuildScript/build-installer.py on OSX 10.5. Hmm. When I tried it (on some 2.5 release), it took me two days until it produced something. Regards, Martin From ronaldoussoren at mac.com Sun Apr 18 17:19:15 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 18 Apr 2010 17:19:15 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BCB228B.2080904@v.loewis.de> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> <4BCB228B.2080904@v.loewis.de> Message-ID: <9F3B36D7-573A-4563-9729-8065AEAF7A18@mac.com> On 18 Apr, 2010, at 17:17, Martin v. L?wis wrote: > Ronald Oussoren wrote: >> On 15 Apr, 2010, at 6:36, Martin v. L?wis wrote: >> >>> Greg Ewing wrote: >>>> Michael Foord wrote: >>>>> Building Python requires, I believe, the XCode development tools to be >>>>> installed. Even then, building a full version of Python - with *all* >>>>> the C extensions that are part of a Python release - is not a trivial >>>>> task. >>>> What's non-trivial about it? >>> Building a DMG file, in a way that the output will actually work on most >>> other systems. >> >> That *is* trivial: use Mac/BuildScript/build-installer.py on OSX 10.5. > > Hmm. When I tried it (on some 2.5 release), it took me two days until it > produced something. It should be much improved in 2.6 and later, please file a bug if it doesn't work for you. Ronald > > Regards, > Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From martin at v.loewis.de Sun Apr 18 17:20:41 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 18 Apr 2010 17:20:41 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> <4BC7F70A.9000503@v.loewis.de> Message-ID: <4BCB2349.8080200@v.loewis.de> > We can certainly start by just trying to automate the execution of > build-installer, something I suspect can be completely controlled from > the master side, and then uploading the resulting dmg file. I'd be > happy to help coordinate any experiments offline if you're interested. Sure! > I do currently have a DMG built for 2.7 Beta 1, if it would be useful. As I said before: if you would plan to do this on a regular basis, for all upcoming releases, that would certainly be a good thing. Having just a single DMG file for some beta release doesn't really help, as testing results for that binary might not transfer to the next one (if the next one comes from a different environment again). Regards, Martin From ronaldoussoren at mac.com Sun Apr 18 17:50:18 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 18 Apr 2010 17:50:18 +0200 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) In-Reply-To: <4BCB2349.8080200@v.loewis.de> References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> <4BC7F70A.9000503@v.loewis.de> <4BCB2349.8080200@v.loewis.de> Message-ID: <9B138233-5FE5-4F5A-A79C-6A530516C129@mac.com> On 18 Apr, 2010, at 17:20, Martin v. L?wis wrote: >> We can certainly start by just trying to automate the execution of >> build-installer, something I suspect can be completely controlled from >> the master side, and then uploading the resulting dmg file. I'd be >> happy to help coordinate any experiments offline if you're interested. > > Sure! > >> I do currently have a DMG built for 2.7 Beta 1, if it would be useful. > > As I said before: if you would plan to do this on a regular basis, for > all upcoming releases, that would certainly be a good thing. > > Having just a single DMG file for some beta release doesn't really help, > as testing results for that binary might not transfer to the next one > (if the next one comes from a different environment again). I will provide installers for 2.7 starting from beta2, Ronald > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From brian.curtin at gmail.com Sun Apr 18 19:37:49 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sun, 18 Apr 2010 12:37:49 -0500 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: <20100418151652.2682.989910686.divmod.xquotient.4@localhost.localdomain> References: <20100418151652.2682.989910686.divmod.xquotient.4@localhost.localdomain> Message-ID: On Sun, Apr 18, 2010 at 10:16, wrote: > On 02:56 pm, techtonik at gmail.com wrote: > >> Twisted folks will surely appreciate any help and may be able to >> contribute back. >> http://twistedmatrix.com/trac/wiki/Windows >> > > Extra Windows and VS licenses would certainly be helpful for Twisted > development, and might lead indirectly to CPython/Windows improvements. Is > this the kind of use for which it is appropriate to request an MSDN license > via the PSF? > > Jean-Paul > To my knowledge, no. I'm only handling PSF-specific subscriptions so I wouldn't be comfortable giving them out for developers of Twisted or any other project. However, I would encourage those projects to contact the OSTC directly to start up a relationship. Since you are active and fit the criteria for CPython, I can give you a subscription and I know of no restrictions on what it's used for outside of CPython. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From florent.xicluna at gmail.com Sun Apr 18 20:07:12 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Sun, 18 Apr 2010 20:07:12 +0200 Subject: [Python-Dev] bbreport In-Reply-To: References: <201004172041.10219.victor.stinner@haypocalc.com> Message-ID: > Does it really need trunk? ?I've been running it under 2.6 without > problems, but I probably haven't explored all the options fully. > Now it works on 2.6 (or 2.5+pysqlite). Regards, -- Florent From steven.bethard at gmail.com Sun Apr 18 21:33:28 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 12:33:28 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCADF66.8070702@gmx.de> References: <4BCADF66.8070702@gmx.de> Message-ID: On Sun, Apr 18, 2010 at 3:31 AM, Tobias Herp wrote: > To put it short: > *Argparse should simply do this like optparse does already.* > I.e., use '--version', '--help' and '-h' by default, but not '-v'. [snip] > What happened was the following: > Completely unnecessarily, the 'version' constructor argument is now > deprecated. [snip] > The deprecation of the 'version' argument press-gangs people to replace > it by > ?parser.add_argument('--version', action='version', > ? ? ? ? ? ? ? ? ? ? ?version='', ? ?# the only common part > ? ? ? ? ? ? ? ? ? ? ?help="show program's version number and exit") What Tobias has conveniently omitted is that there was not much agreement on what the default behavior of the version flag should be. Things that were proposed: * Use both -v and --version * Use both -V and --version * Use just --version * Print out just the version string * Print out program name with the version string I put up a poll and asked people for feedback on some of the alternatives: http://www.vizu.com/res/Grab-bag/argparse/poll-results.html?n=192933 And it was far from a consensus. Given the lack of consensus, the many different options people thought were the "right" way to do things, and the fact that the action='version' approach allows people to customize the flags to whatever they want, I stick by my decision to deprecate the constructor argument and redirect people to the more flexible add_argument approach. In the face of ambiguity, refuse the temptation to guess. In general, I should say that I'm not opposed to changing the behavior in Python trunk, even if it causes a bit of backwards incompatibility. But I'd really like a consensus about the correct behavior, and so far I have not seen that. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Sun Apr 18 21:38:10 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 12:38:10 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCAE706.2090503@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCAE706.2090503@gmail.com> Message-ID: On Sun, Apr 18, 2010 at 4:03 AM, Nick Coghlan wrote: > To deal with this in a backwards compatible way while remaining on the > path to more conventional behaviour, I suggest the following: > > 1. For Python 2.7, deprecate *just* the "-v" default behaviour for the > version. This means "--version" and "-v" will be set to invoke different > actions when the version argument is supplied (the latter will trigger a > deprecation warning if supplied, while the former will work normally). It's not clear how you would do this. If you can suggest a patch, I'd be happy to consider it. However, and the moment, you get "-v" and "--version" by simply specifying ArgumentParser(..., version="XXX"). So how do you deprecate just the "-v"? All I can imagine you mean is to issue a deprecation warning whenever a user of the script provides "-v" at the command line, but that seems sketchy to me - we'd be deprecating features of someone's *application*, not features of the argparse *library*. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From db3l.net at gmail.com Sun Apr 18 22:55:13 2010 From: db3l.net at gmail.com (David Bolen) Date: Sun, 18 Apr 2010 16:55:13 -0400 Subject: [Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) References: <155072217312148274731633653972868664038-Webmail@me.com> <42735872743376525492939197280196737792-Webmail@me.com> <4BC77BB1.3070106@v.loewis.de> <4BC7F70A.9000503@v.loewis.de> <4BCB2349.8080200@v.loewis.de> Message-ID: "Martin v. L?wis" writes: >> I do currently have a DMG built for 2.7 Beta 1, if it would be useful. > > As I said before: if you would plan to do this on a regular basis, for > all upcoming releases, that would certainly be a good thing. No argument - just figured I'd offer to get past the near term resource issue, presuming that having a beta installer available is better than not, even with some build-environment difference risk. I could probably commit to a longer term too, but it sounds like Ronald is still on board for that. I've also been a little hesitant since there's clearly a lot of effort that has been put into the Mac build stuff and I didn't want to be the inexperienced bull in the china shop compared to those who have been doing it for a while. I'm happy to stick with the buildbot side of things for now, and remain willing to assist in any changes to better match the buildbot builds to the final distribution process for more aligned testing. -- David From db3l.net at gmail.com Sun Apr 18 23:15:18 2010 From: db3l.net at gmail.com (David Bolen) Date: Sun, 18 Apr 2010 17:15:18 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC697D2.4020200@v.loewis.de> <4BCB228B.2080904@v.loewis.de> <9F3B36D7-573A-4563-9729-8065AEAF7A18@mac.com> Message-ID: Ronald Oussoren writes: > On 18 Apr, 2010, at 17:17, Martin v. L?wis wrote: > >> Ronald Oussoren wrote: >>> >>> That *is* trivial: use Mac/BuildScript/build-installer.py on OSX 10.5. >> >> Hmm. When I tried it (on some 2.5 release), it took me two days until it >> produced something. > > It should be much improved in 2.6 and later, please file a bug if it > doesn't work for you. For what it's worth, the trunk currently takes ~25 minutes on my relatively modest Mini, when run in parallel with some buildbot stuff, and with third-party sources already downloaded. More or less evenly split among third-party packages, the interpreter itself, and then docs/framework/disk image creation. -- David From Tobias.Herp at gmx.de Sun Apr 18 23:35:06 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Sun, 18 Apr 2010 23:35:06 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> Message-ID: <4BCB7B0A.6000305@gmx.de> Steven Bethard schrieb: > On Sun, Apr 18, 2010 at 3:31 AM, Tobias Herp wrote: >> To put it short: >> *Argparse should simply do this like optparse does already.* >> I.e., use '--version', '--help' and '-h' by default, but not '-v'. > [snip] >> What happened was the following: >> Completely unnecessarily, the 'version' constructor argument is now >> deprecated. > [snip] >> The deprecation of the 'version' argument press-gangs people to replace >> it by >> parser.add_argument('--version', action='version', >> version='', # the only common part >> help="show program's version number and exit") > > What Tobias has conveniently omitted is that there was not much > agreement on what the default behavior of the version flag should be. Well, the mail was quite long already, and the issue-tracker issue was linked. > Things that were proposed: > > * Use both -v and --version This is very unusual. I don't know of a single example other than existing argparse. Incompatible to existing optparse usage. > * Use both -V and --version A very common combination; many programs which support --version accompany it with -V. However, if we'd add '-V' to the default option strings, we would likely break existing programs which worked around the so-far default. > * Use just --version IMO the only way, and the way optparse does it. > * Print out just the version string > * Print out program name with the version string If this is an issue: With optparse, the version argument value is printed, which is often something like '%prog '+VERSION (for argparse, '%(prog)s '+VERSION or '%%(prog)s %s' % VERSION). This way, the decision is left to the programmer. Completely agreeable, as far as I'm concerned. > I put up a poll [...] > > Given the lack of consensus, the many > different options people thought were the "right" way to do things, > and the fact that the action='version' approach allows people to > customize the flags to whatever they want, I stick by my decision to > deprecate the constructor argument ... A. Very. Bad. Idea. > ... and redirect people to the more flexible add_argument approach. It is ok to /allow/ people to customize the flags to whatever they want. It is /not/ ok to /force/ them to do so, if there is a perfectly reasonable solution which allows them to stick with their habits. And there is one: drop the '-v' default. The only way (see above). > In the face of ambiguity, refuse the temptation to guess. No guessing is needed (and the ZoP aphorism is not applicable here: this is not about guessing what the user might have meant. Besides, if the user specifies '--version', [s]he /very/ likely wants some info about the program version; giving '-v' can have been done looking forward to --verbose output (ok, nothing bad will happen, but it mismatches user expectations). > In general, I should say that I'm not opposed to changing the behavior > in Python trunk, even if it causes a bit of backwards incompatibility. A /very little/ bit. When fetching stdout to get the program version, it is much more reasonable to rely on '--version' than on '-v'. This is /the/ option string which is best supported by far. > But I'd really like a consensus about the correct behavior, and so far > I have not seen that. We'll have one ;-) -- Tobias From Tobias.Herp at gmx.de Sun Apr 18 23:48:55 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Sun, 18 Apr 2010 23:48:55 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCAE706.2090503@gmail.com> Message-ID: <4BCB7E47.5010800@gmx.de> Steven Bethard schrieb: > On Sun, Apr 18, 2010 at 4:03 AM, Nick Coghlan wrote: >> To deal with this in a backwards compatible way while remaining on the >> path to more conventional behaviour, I suggest the following: >> >> 1. For Python 2.7, deprecate *just* the "-v" default behaviour for the >> version. This means "--version" and "-v" will be set to invoke different >> actions when the version argument is supplied (the latter will trigger a >> deprecation warning if supplied, while the former will work normally). > > [...] > > All I can imagine you mean is to issue a deprecation warning whenever > a user of the script provides "-v" at the command line, but that seems > sketchy to me - we'd be deprecating features of someone's > *application*, not features of the argparse *library*. It would raise warnings when the option is /used/ (rather than defined; such warnings address programmers, though), and it wouldn't free '-v' for other uses. I agree that this would be more complicated than necessary. It would be better to just drop the default usage of '-v'. This way we wouldn't need a deprecation procedure either. -- Tobias From victor.stinner at haypocalc.com Sun Apr 18 23:57:03 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 18 Apr 2010 23:57:03 +0200 Subject: [Python-Dev] bbreport In-Reply-To: <201004172041.10219.victor.stinner@haypocalc.com> References: <201004172041.10219.victor.stinner@haypocalc.com> Message-ID: <201004182357.03376.victor.stinner@haypocalc.com> Le samedi 17 avril 2010 20:41:10, Victor Stinner a ?crit : > Ezio and Florent are developing a tool called bbreport to collect buildbot > results and generate short reports to the command line. (...) I realized that most issues come from Windows and Mac. Can't we just turn off these buildbots? -- Victor Stinner http://www.haypocalc.com/ From steven.bethard at gmail.com Mon Apr 19 00:46:00 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 15:46:00 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB7B0A.6000305@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: > Steven Bethard schrieb: >> On Sun, Apr 18, 2010 at 3:31 AM, Tobias Herp wrote: >>> *Argparse should simply do this like optparse does already.* >>> I.e., use '--version', '--help' and '-h' by default, but not '-v'. >> [snip] >>> The deprecation of the 'version' argument press-gangs people to replace >>> it by >>> ?parser.add_argument('--version', action='version', >>> ? ? ? ? ? ? ? ? ? ? ?version='', ? ?# the only common part >>> ? ? ? ? ? ? ? ? ? ? ?help="show program's version number and exit") >> >> What Tobias has conveniently omitted is that there was not much >> agreement on what the default behavior of the version flag should be. [snip a bunch of Tobias's opinions on what the default should be] >> >> But I'd really like a consensus about the correct behavior, and so far >> I have not seen that. > > We'll have one ;-) I hope you can understand that I don't consider your personal opinion alone as a consensus. As I said, I'm willing to change the defaults and even break backwards compatibility a bit in Python trunk[1] but I need to a see a consensus from a variety of developers that "--version" is the right answer, and not "-V/--version", etc. Note that even though I agree with you that "-v/--version" is probably not the best choice, in the poll[2] 11% of people still wanted this. So it's not a simple decision. By the way, we could simplify the typical add_argument usage by adding "show program's version number and exit" as the default help for the 'version' action. Then you should just write: parser.add_argument('--version', action='version', version='') Steve [1] Assuming the release manager will allow it. [2] http://www.vizu.com/res/Grab-bag/argparse/poll-results.html?n=192933 -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From solipsis at pitrou.net Mon Apr 19 00:52:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Apr 2010 22:52:12 +0000 (UTC) Subject: [Python-Dev] Python 2.7b1 and argparse's version action References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: Steven Bethard gmail.com> writes: > > I > need to a see a consensus from a variety of developers that > "--version" is the right answer, and not "-V/--version", etc. Both are ok for me. "-v" as a shortcut for "--version" looks wrong, though. "-v" is almost always used for verbosity these days. > Note > that even though I agree with you that "-v/--version" is probably not > the best choice, in the poll[2] 11% of people still wanted this. This strikes me as a small minority. Regards Antoine. From ncoghlan at gmail.com Mon Apr 19 00:57:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Apr 2010 08:57:24 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: <4BCB8E54.3040806@gmail.com> Steven Bethard wrote: > By the way, we could simplify the typical add_argument usage by adding > "show program's version number and exit" as the default help for the > 'version' action. Then you should just write: > > parser.add_argument('--version', action='version', version='') With that change, I would have no problem with the current argparse behaviour (since doing it this way makes it very easy for people to add a "-V" shortcut if they want one). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Mon Apr 19 00:57:54 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 19 Apr 2010 00:57:54 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: <4BCB8E72.7060608@voidspace.org.uk> On 19/04/2010 00:52, Antoine Pitrou wrote: > Steven Bethard gmail.com> writes: > >> I >> need to a see a consensus from a variety of developers that >> "--version" is the right answer, and not "-V/--version", etc. >> > Both are ok for me. "-v" as a shortcut for "--version" looks wrong, though. "-v" > is almost always used for verbosity these days. > Adding -V *could* be incompatible with existing users using it for something else. A default '--version' seems like a sensible solution to me. Michael > >> Note >> that even though I agree with you that "-v/--version" is probably not >> the best choice, in the poll[2] 11% of people still wanted this. >> > This strikes me as a small minority. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From ncoghlan at gmail.com Mon Apr 19 01:04:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Apr 2010 09:04:27 +1000 Subject: [Python-Dev] bbreport In-Reply-To: <201004182357.03376.victor.stinner@haypocalc.com> References: <201004172041.10219.victor.stinner@haypocalc.com> <201004182357.03376.victor.stinner@haypocalc.com> Message-ID: <4BCB8FFB.2020106@gmail.com> Victor Stinner wrote: > Le samedi 17 avril 2010 20:41:10, Victor Stinner a ?crit : >> Ezio and Florent are developing a tool called bbreport to collect buildbot >> results and generate short reports to the command line. (...) > > I realized that most issues come from Windows and Mac. Can't we just turn off > these buildbots? o.O? Remember there's no tone of voice in email... smileys matter when joking :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From steven.bethard at gmail.com Mon Apr 19 01:22:12 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 16:22:12 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB8E54.3040806@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> Message-ID: On Sun, Apr 18, 2010 at 3:57 PM, Nick Coghlan wrote: > Steven Bethard wrote: >> By the way, we could simplify the typical add_argument usage by adding >> "show program's version number and exit" as the default help for the >> 'version' action. Then you should just write: >> >> ? ? parser.add_argument('--version', action='version', version='') > > With that change, I would have no problem with the current argparse > behaviour (since doing it this way makes it very easy for people to add > a "-V" shortcut if they want one). Probably this should happen regardless of the outcome of the constructor argument. The only reason it wasn't already there is that I hadn't thought of it. ;-) Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Mon Apr 19 01:24:59 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 16:24:59 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: On Sun, Apr 18, 2010 at 3:52 PM, Antoine Pitrou wrote: > Steven Bethard gmail.com> writes: >> Note >> that even though I agree with you that "-v/--version" is probably not >> the best choice, in the poll[2] 11% of people still wanted this. > > This strikes me as a small minority. Agreed, but it's also the current behavior, and has been since the beginning of argparse. Note that no one complained about it until Tobias filed the issue in Nov 06, 2009. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From eric at trueblade.com Mon Apr 19 01:30:07 2010 From: eric at trueblade.com (Eric Smith) Date: Sun, 18 Apr 2010 19:30:07 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: <4BCB95FF.8050202@trueblade.com> Steven Bethard wrote: > By the way, we could simplify the typical add_argument usage by adding > "show program's version number and exit" as the default help for the > 'version' action. Then you should just write: > > parser.add_argument('--version', action='version', version='') I like this the best. I don't like argparse adding arguments for me. Eric. From ncoghlan at gmail.com Mon Apr 19 01:35:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Apr 2010 09:35:41 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> Message-ID: <4BCB974D.7070401@gmail.com> Steven Bethard wrote: > On Sun, Apr 18, 2010 at 3:57 PM, Nick Coghlan wrote: >> Steven Bethard wrote: >>> By the way, we could simplify the typical add_argument usage by adding >>> "show program's version number and exit" as the default help for the >>> 'version' action. Then you should just write: >>> >>> parser.add_argument('--version', action='version', version='') >> With that change, I would have no problem with the current argparse >> behaviour (since doing it this way makes it very easy for people to add >> a "-V" shortcut if they want one). > > Probably this should happen regardless of the outcome of the > constructor argument. The only reason it wasn't already there is that > I hadn't thought of it. ;-) Crazy thought... would it make sense to have the following implicitly use "--version" as the option flag: parser.add_argument(action='version', version='
') There are two things about the explicit '--version' that bother me: 1. It reduces the automatic provision of "standard" option spellings 2. The repetition in reading/writing 'version' 3 times is kind of annoying (Probably a bad idea, since adding "-V" would mean having to add "--version" as well, but figured it was worth mentioning). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From Tobias.Herp at gmx.de Mon Apr 19 02:07:35 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Mon, 19 Apr 2010 02:07:35 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB95FF.8050202@trueblade.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> Message-ID: <4BCB9EC7.9030309@gmx.de> Eric Smith schrieb: > Steven Bethard wrote: >> By the way, we could simplify the typical add_argument usage by adding >> "show program's version number and exit" as the default help for the >> 'version' action. Then you should just write: >> >> parser.add_argument('--version', action='version', version='> version>') > > I like this the best. I don't like argparse adding arguments for me. There is no reason why you shouldn't get it. You could always (and with optparse as well as argparse) omit the version argument and build or build not the version argument yourself. Or a special add_version_option (hey, it's really an /option/) or add_version_argument method, which would work without if statements and could be overridden in subclasses. But note that - many optparse programs use the version argument - many other programmers find this feature very convenient - dropping or deprecating this is a totally unnecessary change (I have not read a single real reason /why/ this should be done). -- Tobias From Tobias.Herp at gmx.de Mon Apr 19 02:09:47 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Mon, 19 Apr 2010 02:09:47 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: <4BCB9F4B.8080309@gmx.de> Steven Bethard schrieb: > On Sun, Apr 18, 2010 at 3:52 PM, Antoine Pitrou wrote: >> Steven Bethard gmail.com> writes: >>> Note >>> that even though I agree with you that "-v/--version" is probably not >>> the best choice, in the poll[2] 11% of people still wanted this. >> >> This strikes me as a small minority. > > Agreed, but it's also the current behavior, ... The current /broken, non-standard/ behaviour. For keeping a current status quo, this is an even worse poll. Very likely some of these voted for this choice because they were afraid of changes (unnecessarily, as explained before). > ... and has been since the beginning of argparse. It's not my fault that in the beginning apparently nobody cared about existing conventions. > Note that no one complained about it until > Tobias filed the issue in Nov 06, 2009. This is not an argument at all. I have collected lots of examples of widely used programs and their options. The only example of '-v', '--verbose' is argparse. It is vital we meet user expectations, and that we support developers in producing programs that do so. -- Tobias From Tobias.Herp at gmx.de Mon Apr 19 03:02:32 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Mon, 19 Apr 2010 03:02:32 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: <4BCBABA8.4080704@gmx.de> Steven Bethard schrieb: > On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: >> Steven Bethard schrieb: >>> On Sun, Apr 18, 2010 at 3:31 AM, Tobias Herp wrote: >>>> *Argparse should simply do this like optparse does already.* >>>> I.e., use '--version', '--help' and '-h' by default, but not '-v'. >>> [snip] >>>> The deprecation of the 'version' argument press-gangs people to replace >>>> it by >>>> parser.add_argument('--version', action='version', >>>> version='', # the only common part >>>> help="show program's version number and exit") >>> >>> What Tobias has conveniently omitted is that there was not much >>> agreement on what the default behavior of the version flag should be. > [snip a bunch of Tobias's opinions on what the default should be] >>> >>> But I'd really like a consensus about the correct behavior, and so far >>> I have not seen that. >> >> We'll have one ;-) > > I hope you can understand that I don't consider your personal opinion > alone as a consensus. Be careful. There have been other personal opinions, and there are very many *X programs which form a standard which should be followed, and this is a /very strong/ argument, IMO. What you conveniently omitted is that I collected a lot of examples and commented the choices (part of which were part of your poll). Unfortunately you completely ignored these very prosaic arguments. By the way, I just had a look at your poll (currently 81 votes). Correct me if I'm wrong, but... - AFAICS, you didn't provide your audience with any background information, e.g. - how optparse does it (and migration should be as easy as possible, right?) - the existing de-facto standard "--version [-V]" - the choice which reflects my proposal got the most votes (46.9% currently) - 42.0% voted for dropping the version argument, which is a minority (bzw., do you really think the 46.9% would like the version argument to be deprecated?!) which apparently didn't consider the optparse migration/analogy issue. Furthermore, nobody who dislikes the version feature is forced to use it. If 60% had voted for removal of a feature which is important and convenient for 40%, would you remove it? Really? IMO, removal of a feature which is used by 40% is out of the question; and if removal is, deprecation is as well. Back to the actual numbers: dropping the argument is impossible (/and so is deprecation/); drop the 42.0%. Take the remaining numbers: 46.9% voted for the change, and 11.1% against it. Do you take your own poll seriously? -- Tobias From ncoghlan at gmail.com Mon Apr 19 03:40:53 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Apr 2010 11:40:53 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCBABA8.4080704@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> Message-ID: <4BCBB4A5.1090401@gmail.com> Tobias Herp wrote: > Be careful. There have been other personal opinions, and there are very > many *X programs which form a standard which should be followed, and > this is a /very strong/ argument, IMO. If the "version=" argument were to only provide "--version", how would you conveniently add a "-V" shorthand? This is trivial with the add_argument approach, but less straightforward with a "version=" argument to the parser constructor. People that want "-v" to mean verbosity in 2.x can also trivially pass "conflict_handler='resolve'" to override the implicitly created version shorthand. Would it be more convenient if argparse had never deviated from the GNU and optparse command line conventions in this respect? Sure it would be. But given that it did, a 'version' action that makes it trivial to choose the details of your version flags is a decent alternative approach to a now-ambiguous version argument to the parser constructor (especially once the default help string for the option is added). Note there are two changes I believe should be made to the argparse documentation for 2.7 though: - the '--version' example should either not use a shorthand, or should use the conventional '-V' - this issue needs to be mentioned in the section on migrating from optparse. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From cs at zip.com.au Mon Apr 19 03:41:08 2010 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 19 Apr 2010 11:41:08 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: Message-ID: <20100419014108.GA26874@cskk.homeip.net> On 18Apr2010 22:52, Antoine Pitrou wrote: | Steven Bethard gmail.com> writes: | > I | > need to a see a consensus from a variety of developers that | > "--version" is the right answer, and not "-V/--version", etc. | | Both are ok for me. "-v" as a shortcut for "--version" looks wrong, though. | "-v" is almost always used for verbosity these days. My view is the same. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ By dint of railing at idiots one runs the risk of becoming idiotic oneself. - Gustave Flaubert From steve at holdenweb.com Mon Apr 19 03:44:54 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 18 Apr 2010 21:44:54 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCBABA8.4080704@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> Message-ID: <4BCBB596.9020806@holdenweb.com> Tobias Herp wrote: > Steven Bethard schrieb: >> On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: >>> Steven Bethard schrieb: >>>> On Sun, Apr 18, 2010 at 3:31 AM, Tobias Herp wrote: >>>>> *Argparse should simply do this like optparse does already.* >>>>> I.e., use '--version', '--help' and '-h' by default, but not '-v'. >>>> [snip] >>>>> The deprecation of the 'version' argument press-gangs people to replace >>>>> it by >>>>> parser.add_argument('--version', action='version', >>>>> version='', # the only common part >>>>> help="show program's version number and exit") >>>> What Tobias has conveniently omitted is that there was not much >>>> agreement on what the default behavior of the version flag should be. >> [snip a bunch of Tobias's opinions on what the default should be] >>>> But I'd really like a consensus about the correct behavior, and so far >>>> I have not seen that. >>> We'll have one ;-) >> I hope you can understand that I don't consider your personal opinion >> alone as a consensus. > > Be careful. There have been other personal opinions, and there are very > many *X programs which form a standard which should be followed, and > this is a /very strong/ argument, IMO. > > What you conveniently omitted is that I collected a lot of examples and > commented the choices (part of which were part of your poll). > Unfortunately you completely ignored these very prosaic arguments. > > By the way, I just had a look at your poll (currently 81 votes). > Correct me if I'm wrong, but... > - AFAICS, you didn't provide your audience with any background > information, e.g. > - how optparse does it (and migration should be as easy as possible, > right?) > - the existing de-facto standard "--version [-V]" > - the choice which reflects my proposal got the most votes > (46.9% currently) > - 42.0% voted for dropping the version argument, which is a minority > (bzw., do you really think the 46.9% > would like the version argument to be deprecated?!) > which apparently didn't consider the optparse migration/analogy issue. > > Furthermore, nobody who dislikes the version feature is forced to use it. > If 60% had voted for removal of a feature which is important and > convenient for 40%, would you remove it? Really? > IMO, removal of a feature which is used by 40% is out of the question; > and if removal is, deprecation is as well. > > Back to the actual numbers: dropping the argument is impossible (/and so > is deprecation/); drop the 42.0%. > Take the remaining numbers: 46.9% voted for the change, and 11.1% > against it. > > Do you take your own poll seriously? > When was this ever a democracy? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From ianb at colorstudy.com Mon Apr 19 04:02:17 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 18 Apr 2010 21:02:17 -0500 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: On Sun, Apr 18, 2010 at 6:24 PM, Steven Bethard wrote: > On Sun, Apr 18, 2010 at 3:52 PM, Antoine Pitrou > wrote: > > Steven Bethard gmail.com> writes: > >> Note > >> that even though I agree with you that "-v/--version" is probably not > >> the best choice, in the poll[2] 11% of people still wanted this. > > > > This strikes me as a small minority. > > Agreed, but it's also the current behavior, and has been since the > beginning of argparse. Note that no one complained about it until > Tobias filed the issue in Nov 06, 2009. > I encountered this problem within minutes of first using argparse. Of course I'm very familiar with optparse and the standard optparse instantiation flies off my fingers without thinking. But then there's going to be a lot more people with that background using argparse once it is in the standard library -- people who don't really care about argparse or optparse but just want to use the standard thing. I don't see any reason why argparse can't simply do exactly what optparse did. There's nothing wrong with it. It's what many people expect. We should just defer to tradition when the choice isn't important (it's getting to be a very bike shed thread). Somewhat relatedly, what is the plan for past and future argparse releases? Michael Foord for instance is releasing unittest improvements in parallel under the name unittest2. I believe there is strong disfavor with releasing packages that overlap with the standard library, so continuing to release argparse under the name argparse will cause problems. I would hate to see release complications or confusions keep argparse from seeing future development. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Apr 19 04:19:46 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 18 Apr 2010 22:19:46 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB95FF.8050202@trueblade.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> Message-ID: <20100418221946.39d6160c@heresy> On Apr 18, 2010, at 07:30 PM, Eric Smith wrote: >Steven Bethard wrote: >> By the way, we could simplify the typical add_argument usage by adding >> "show program's version number and exit" as the default help for the >> 'version' action. Then you should just write: >> >> parser.add_argument('--version', action='version', version='') > >I like this the best. I don't like argparse adding arguments for me. I concur. This gives me all the flexibility I need to make my programs accept exactly the arguments I want and print exactly the information I want to present. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ben+python at benfinney.id.au Mon Apr 19 04:38:21 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 19 Apr 2010 12:38:21 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB9F4B.8080309@gmx.de> Message-ID: <87d3xwi2ua.fsf@benfinney.id.au> Tobias Herp writes: > Steven Bethard schrieb: > > On Sun, Apr 18, 2010 at 3:52 PM, Antoine Pitrou wrote: > >> Steven Bethard gmail.com> writes: > >>> Note that even though I agree with you that "-v/--version" is > >>> probably not the best choice, in the poll[2] 11% of people still > >>> wanted this. > >> > >> This strikes me as a small minority. > > > > Agreed, but it's also the current behavior, ... > > The current /broken, non-standard/ behaviour. I think that is overstating your case. Steven is pointing out the fact that ?default to ?-v? and ?--version?? is the current behaviour for argparse. It's also a fact that argparse, as a third-party library, has an existing user base, and changing this behaviour incompatibly is not something to do lightly. To call this behaviour ?broken? is a stretch too far, IMO. It does break compatibility with optparse, but it's not *inherently* broken as the above would imply. Nor is there really a standard for this behaviour to be measured ?non-standard?. At best, there is a strong convention among Unix command-line programs of ?--version? for a version action and ?-v? for enabling verbose output. But that convention is not normative, so it's too much to call it a standard IMO. > I have collected lots of examples of widely used programs and their > options. The only example of '-v', '--verbose' is argparse. I'm not sure what programs you're talking about (argparse is a library, not a program that can be run by itself), but in a brief search of my system I've found several programs that use ?-v? for a version action, including among others: * Info-ZIP's ?zip? and ?unzip? * GNU ?fdisk? * Vim's ?xxd? * W3C's ?tidy? These are certainly widely deployed, although they may not be widely familiar. I do agree that such programs are a small minority, but it's overstating your case to call them ?non-standard?, since there's no standard for this, or ?broken?, since they demonstrably work fine. Rather, they buck what is certainly a firm convention to avoid ?-v? for a version option. I'm certainly in favour of preserving that convention by avoiding use of ?-v? for version. I would rather, though, that the discussion remain aware of facts and not descend into hyperbole. -- \ ?Perchance you who pronounce my sentence are in greater fear | `\ than I who receive it.? ?Giordano Bruno, burned at the stake by | _o__) the Catholic church for the heresy of heliocentrism, 1600-02-16 | Ben Finney From steve at holdenweb.com Mon Apr 19 05:34:51 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 18 Apr 2010 23:34:51 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <20100419014108.GA26874@cskk.homeip.net> References: <20100419014108.GA26874@cskk.homeip.net> Message-ID: <4BCBCF5B.8060902@holdenweb.com> Cameron Simpson wrote: > On 18Apr2010 22:52, Antoine Pitrou wrote: > | Steven Bethard gmail.com> writes: > | > I > | > need to a see a consensus from a variety of developers that > | > "--version" is the right answer, and not "-V/--version", etc. > | > | Both are ok for me. "-v" as a shortcut for "--version" looks wrong, though. > | "-v" is almost always used for verbosity these days. > > My view is the same. I don't expect -v to be one or the other, and I certainly don't expect the rest of the world to want to confirm to what the Python community decides. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Mon Apr 19 05:34:51 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 18 Apr 2010 23:34:51 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <20100419014108.GA26874@cskk.homeip.net> References: <20100419014108.GA26874@cskk.homeip.net> Message-ID: <4BCBCF5B.8060902@holdenweb.com> Cameron Simpson wrote: > On 18Apr2010 22:52, Antoine Pitrou wrote: > | Steven Bethard gmail.com> writes: > | > I > | > need to a see a consensus from a variety of developers that > | > "--version" is the right answer, and not "-V/--version", etc. > | > | Both are ok for me. "-v" as a shortcut for "--version" looks wrong, though. > | "-v" is almost always used for verbosity these days. > > My view is the same. I don't expect -v to be one or the other, and I certainly don't expect the rest of the world to want to confirm to what the Python community decides. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From rrr at ronadam.com Mon Apr 19 05:44:29 2010 From: rrr at ronadam.com (Ron Adam) Date: Sun, 18 Apr 2010 22:44:29 -0500 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB8E54.3040806@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> Message-ID: On 04/18/2010 05:57 PM, Nick Coghlan wrote: > Steven Bethard wrote: >> By the way, we could simplify the typical add_argument usage by adding >> "show program's version number and exit" as the default help for the >> 'version' action. Then you should just write: >> >> parser.add_argument('--version', action='version', version='') > > With that change, I would have no problem with the current argparse > behaviour (since doing it this way makes it very easy for people to add > a "-V" shortcut if they want one). +1 This sounds good to me also. Note that the python interpreter uses -V and --version. ra at Gutsy:~$ python3.1 -V Python 3.1.2 ra at Gutsy:~$ python3.1 --version Python 3.1.2 And -v is used as follows: -v : verbose (trace import statements); also PYTHONVERBOSE=x can be supplied multiple times to increase verbosity Ron From rrr at ronadam.com Mon Apr 19 06:05:41 2010 From: rrr at ronadam.com (Ron Adam) Date: Sun, 18 Apr 2010 23:05:41 -0500 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB974D.7070401@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> <4BCB974D.7070401@gmail.com> Message-ID: On 04/18/2010 06:35 PM, Nick Coghlan wrote: > Steven Bethard wrote: >> On Sun, Apr 18, 2010 at 3:57 PM, Nick Coghlan wrote: >>> Steven Bethard wrote: >>>> By the way, we could simplify the typical add_argument usage by adding >>>> "show program's version number and exit" as the default help for the >>>> 'version' action. Then you should just write: >>>> >>>> parser.add_argument('--version', action='version', version='') >>> With that change, I would have no problem with the current argparse >>> behaviour (since doing it this way makes it very easy for people to add >>> a "-V" shortcut if they want one). >> >> Probably this should happen regardless of the outcome of the >> constructor argument. The only reason it wasn't already there is that >> I hadn't thought of it. ;-) > > Crazy thought... would it make sense to have the following implicitly > use "--version" as the option flag: > > parser.add_argument(action='version', version='
') > > There are two things about the explicit '--version' that bother me: > 1. It reduces the automatic provision of "standard" option spellings I think any non standard spellings will either be on purpose or be caught fairly quickly. And in either case I can't imagine it having an impact on the rest of the program, so I wouldn't worry about this too much. > 2. The repetition in reading/writing 'version' 3 times is kind of annoying Ahh, but it isn't the same exact 'version' each time. One is an input specifier string, one is an action key, and the last is a value name. > (Probably a bad idea, since adding "-V" would mean having to add > "--version" as well, but figured it was worth mentioning). I agree. Even though it may seem redundant when writing it, it's both clear and explicit when reading it even if you aren't very familiar with how argparse works, or have just returned from a really long and fun vacation. ;-) Ron From martin at v.loewis.de Mon Apr 19 06:21:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Apr 2010 06:21:55 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB9EC7.9030309@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <4BCB9EC7.9030309@gmx.de> Message-ID: <4BCBDA63.5080103@v.loewis.de> > - many optparse programs use the version argument > - many other programmers find this feature very convenient > - dropping or deprecating this is a totally unnecessary change > (I have not read a single real reason /why/ this should be done). You actually brought up a good reason yourself: In the face of ambiguity, refuse the temptation to guess. If you ask "give me a version argument", the question is "how is it spelled?". IIUC, you originally complained that the spelling of argparse (i.e. -v/--version) is not good, and that a different spelling should be used. So it's ambiguous, in which case the feature shouldn't be provided in the first place. Regards, Martin From steven.bethard at gmail.com Mon Apr 19 06:22:06 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 21:22:06 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCB974D.7070401@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> <4BCB974D.7070401@gmail.com> Message-ID: On Sun, Apr 18, 2010 at 4:35 PM, Nick Coghlan wrote: > Steven Bethard wrote: >> On Sun, Apr 18, 2010 at 3:57 PM, Nick Coghlan wrote: >>> Steven Bethard wrote: >>>> By the way, we could simplify the typical add_argument usage by adding >>>> "show program's version number and exit" as the default help for the >>>> 'version' action. Then you should just write: >>>> >>>> ? ? parser.add_argument('--version', action='version', version='') >>> With that change, I would have no problem with the current argparse >>> behaviour (since doing it this way makes it very easy for people to add >>> a "-V" shortcut if they want one). >> >> Probably this should happen regardless of the outcome of the >> constructor argument. The only reason it wasn't already there is that >> I hadn't thought of it. ;-) > > Crazy thought... would it make sense to have the following implicitly > use "--version" as the option flag: > > ?parser.add_argument(action='version', version='
') It would be possible, but not simple. The main drawback is that it would have to be done in the parser code, and not in the VersionAction class where all the other version behavior is. I think it's not a huge gain over supplying the option strings, particularly if you're following, say, the Python executable conventions: parser.add_argument('-V', action='version', version='
') Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Mon Apr 19 06:40:45 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 21:40:45 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCBB4A5.1090401@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> <4BCBB4A5.1090401@gmail.com> Message-ID: On Sun, Apr 18, 2010 at 6:40 PM, Nick Coghlan wrote: > Note there are two changes I believe should be made to the argparse > documentation for 2.7 though: > - the '--version' example should either not use a shorthand, or should > use the conventional '-V' > - this issue needs to be mentioned in the section on migrating from > optparse. Thanks for pointing these out. I'll make a note to fix these when I fix the default help value. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Mon Apr 19 06:42:53 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 18 Apr 2010 21:42:53 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> Message-ID: On Sun, Apr 18, 2010 at 7:02 PM, Ian Bicking wrote: > Somewhat relatedly, what is the plan for past and future argparse releases? I currently don't have any plans to make releases outside of the main Python releases. Perhaps if there's great demand for it, I'll reconsider, but as it is, I haven't had as much time as I would like for argparse, and managing two different types of releases is probably unrealistic for me at the moment. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From jafo at tummy.com Mon Apr 19 07:54:36 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Sun, 18 Apr 2010 23:54:36 -0600 Subject: [Python-Dev] Bug? syslog.openlog using ident "python" by default. In-Reply-To: <4BA37D1B.9020306@trueblade.com> References: <20100318115844.GA7374@tummy.com> <4BA37D1B.9020306@trueblade.com> Message-ID: <4BCBF01C.3040300@tummy.com> I've implemented this and put the patch in Issue8451: http://bugs.python.org/issue8451 I'm happy with how it is now, including documentation, but would like some review. Thanks, Sean -- Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: From victor.stinner at haypocalc.com Mon Apr 19 11:33:58 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 19 Apr 2010 11:33:58 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? Message-ID: <201004191133.58602.victor.stinner@haypocalc.com> Hi, I'm working on surrogates in filenames on Linux (more generally on BSD and UNIX OS) to support undecodable filenames, see PEP 383. Amaury told me that I only fixed the non-Windows versions (I fixed subprocess about the current directory and _ctypes.dlopen()), but it doesn't work on Windows. It's a choice, I didn't want to patch Windows because I know that Windows use unicode internally. I consider that developers using Python3 should use unicode on Windows, and byte or unicode+surrogates on other OS. I don't know well Windows API, and so I would like your opinion about that ;-) -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Mon Apr 19 12:17:16 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 19 Apr 2010 12:17:16 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: <201004191133.58602.victor.stinner@haypocalc.com> References: <201004191133.58602.victor.stinner@haypocalc.com> Message-ID: <201004191217.16408.victor.stinner@haypocalc.com> Le lundi 19 avril 2010 11:33:58, Victor Stinner a ?crit : > I'm working on surrogates in filenames on Linux (...) Related issues: #8391: os.execvpe() doesn't support surrogates in env #8393: subprocess: support undecodable current working directory on POSIX OS #8394: ctypes.dlopen() doesn't support surrogates #8412: os.system() doesn't support surrogates nor bytes I fixed the 3 last issues (#8393, #8394, #8412) for non-Windows OS. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Mon Apr 19 12:47:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 19 Apr 2010 10:47:00 +0000 (UTC) Subject: [Python-Dev] MSDN licenses available for python-dev References: Message-ID: Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit?: > > The recent threads on builds/installers for Mac and Windows reminded me > of Steve Holden's push to get the python-dev team equipped via a > connection with the Microsoft Open Source Technology Center. The OSTC > team provides Microsoft Developer Network licenses to open source > projects to assist them in better supporting Windows. > > I've talked with Steve (who passed the task to me) and the Microsoft > folks, and they are happy to provide more licenses if needed. If you are > interested in getting a copy of MSDN, please contact me off-list. Does it include a license for Windows itself? Does it allow me to install and run it in a VM? If so, I'm interested. Regards Antoine. From solipsis at pitrou.net Mon Apr 19 13:07:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 19 Apr 2010 11:07:05 +0000 (UTC) Subject: [Python-Dev] Support byte string API of Windows in Python3? References: <201004191133.58602.victor.stinner@haypocalc.com> Message-ID: Victor Stinner haypocalc.com> writes: > > It's a choice, I didn't want to patch Windows because I know that Windows use > unicode internally. I consider that developers using Python3 should use > unicode on Windows, and byte or unicode+surrogates on other OS. I think both possibilities should be available on all OSes, so as to make it easier to write cross-platform code. Having to switch being bytes and unicode depending on the OS means developers will have to deal with encoding issues themselves, which is suboptimal from a language usability's point of view. Regards Antoine. From steve at holdenweb.com Mon Apr 19 13:48:38 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 19 Apr 2010 07:48:38 -0400 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: <4BCC4316.4040207@holdenweb.com> Antoine Pitrou wrote: > Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : >> The recent threads on builds/installers for Mac and Windows reminded me >> of Steve Holden's push to get the python-dev team equipped via a >> connection with the Microsoft Open Source Technology Center. The OSTC >> team provides Microsoft Developer Network licenses to open source >> projects to assist them in better supporting Windows. >> >> I've talked with Steve (who passed the task to me) and the Microsoft >> folks, and they are happy to provide more licenses if needed. If you are >> interested in getting a copy of MSDN, please contact me off-list. > > Does it include a license for Windows itself? > Does it allow me to install and run it in a VM? > If so, I'm interested. > Yes to both. MSDN offers a very broad license, with activation keys for many products generated on demand. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Mon Apr 19 13:49:27 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 19 Apr 2010 13:49:27 +0200 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: <4BCC4347.9050501@voidspace.org.uk> On 19/04/2010 12:47, Antoine Pitrou wrote: > Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : > >> The recent threads on builds/installers for Mac and Windows reminded me >> of Steve Holden's push to get the python-dev team equipped via a >> connection with the Microsoft Open Source Technology Center. The OSTC >> team provides Microsoft Developer Network licenses to open source >> projects to assist them in better supporting Windows. >> >> I've talked with Steve (who passed the task to me) and the Microsoft >> folks, and they are happy to provide more licenses if needed. If you are >> interested in getting a copy of MSDN, please contact me off-list. >> > Does it include a license for Windows itself? > Does it allow me to install and run it in a VM? > If so, I'm interested. > Yes, MSDN licenses give you access to several versions of Windows plus the Visual Studio tools for compiling Python (amongst other things). Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From p.f.moore at gmail.com Mon Apr 19 14:31:33 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Apr 2010 13:31:33 +0100 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB8E54.3040806@gmail.com> Message-ID: On 19 April 2010 04:44, Ron Adam wrote: > Note that the python interpreter uses -V and --version. > > ra at Gutsy:~$ python3.1 -V > Python 3.1.2 > ra at Gutsy:~$ python3.1 --version > Python 3.1.2 > > And -v is used as follows: > > -v ? ? : verbose (trace import statements); also PYTHONVERBOSE=x > ? ? ? ? can be supplied multiple times to increase verbosity Ironically, I'm forever mistakenly typing "python -v" to get the version. I'm not arguing that this is the "right" behaviour, just pointing out that it's neither unknown, nor entirely surprising to at least one user... As others have pointed out, there's a lot of hyperbole in this thread (not in Ron's post that I quoted, though). If I were using argparse, which I almost certainly will in the future, I'd use the explicit parser.add_argument('--version', action='version', version='') form, so I have no interest in the default version argument - it can be deprecated, removed, kept the same or changed, it makes no difference to me. Paul. From p.f.moore at gmail.com Mon Apr 19 14:33:56 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Apr 2010 13:33:56 +0100 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: On 19 April 2010 11:47, Antoine Pitrou wrote: > Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit?: >> >> The recent threads on builds/installers for Mac and Windows reminded me >> of Steve Holden's push to get the python-dev team equipped via a >> connection with the Microsoft Open Source Technology Center. The OSTC >> team provides Microsoft Developer Network licenses to open source >> projects to assist them in better supporting Windows. >> >> I've talked with Steve (who passed the task to me) and the Microsoft >> folks, and they are happy to provide more licenses if needed. If you are >> interested in getting a copy of MSDN, please contact me off-list. > > Does it include a license for Windows itself? > Does it allow me to install and run it in a VM? > If so, I'm interested. Yes. Arguably that's the best thing about the MSDN licenses (particularly for non-Windows users, but even for us Windows users to set up "clean" VM environments). Paul. From mail at timgolden.me.uk Mon Apr 19 15:00:52 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 19 Apr 2010 14:00:52 +0100 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: Message-ID: <4BCC5404.7020208@timgolden.me.uk> On 19/04/2010 13:33, Paul Moore wrote: > On 19 April 2010 11:47, Antoine Pitrou wrote: >> Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : >>> >>> The recent threads on builds/installers for Mac and Windows reminded me >>> of Steve Holden's push to get the python-dev team equipped via a >>> connection with the Microsoft Open Source Technology Center. The OSTC >>> team provides Microsoft Developer Network licenses to open source >>> projects to assist them in better supporting Windows. >>> >>> I've talked with Steve (who passed the task to me) and the Microsoft >>> folks, and they are happy to provide more licenses if needed. If you are >>> interested in getting a copy of MSDN, please contact me off-list. >> >> Does it include a license for Windows itself? >> Does it allow me to install and run it in a VM? >> If so, I'm interested. > > Yes. Arguably that's the best thing about the MSDN licenses > (particularly for non-Windows users, but even for us Windows users to > set up "clean" VM environments). For some reason I hadn't appreciated that this was the case. I'm in the process of negotiating [*] to repurpose an out-of-work server here as a buildbot and being able to install an O/S will smooth the negotiations considerably. I'm especially keen to get a buildbot running a Windows server OS rather than desktop. TJG [*] May come to nothing; don't hold your breath From brian.curtin at gmail.com Mon Apr 19 15:14:53 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 19 Apr 2010 08:14:53 -0500 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: <4BCC4316.4040207@holdenweb.com> References: <4BCC4316.4040207@holdenweb.com> Message-ID: On Mon, Apr 19, 2010 at 06:48, Steve Holden wrote: > Antoine Pitrou wrote: > > Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : > >> The recent threads on builds/installers for Mac and Windows reminded me > >> of Steve Holden's push to get the python-dev team equipped via a > >> connection with the Microsoft Open Source Technology Center. The OSTC > >> team provides Microsoft Developer Network licenses to open source > >> projects to assist them in better supporting Windows. > >> > >> I've talked with Steve (who passed the task to me) and the Microsoft > >> folks, and they are happy to provide more licenses if needed. If you are > >> interested in getting a copy of MSDN, please contact me off-list. > > > > Does it include a license for Windows itself? > > Does it allow me to install and run it in a VM? > > If so, I'm interested. > > > Yes to both. MSDN offers a very broad license, with activation keys for > many products generated on demand. > The left panel on http://msdn.microsoft.com/en-us/subscriptions/cc137115.aspx has the full list of currently available products if anyone wants to know what's all involved here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Apr 19 16:41:07 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 19 Apr 2010 10:41:07 -0400 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: References: <4BCC4316.4040207@holdenweb.com> Message-ID: <20100419104107.5d9bc674@heresy> On Apr 19, 2010, at 08:14 AM, Brian Curtin wrote: >On Mon, Apr 19, 2010 at 06:48, Steve Holden wrote: > >> Antoine Pitrou wrote: >> > Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : >> >> The recent threads on builds/installers for Mac and Windows reminded me >> >> of Steve Holden's push to get the python-dev team equipped via a >> >> connection with the Microsoft Open Source Technology Center. The OSTC >> >> team provides Microsoft Developer Network licenses to open source >> >> projects to assist them in better supporting Windows. >> >> >> >> I've talked with Steve (who passed the task to me) and the Microsoft >> >> folks, and they are happy to provide more licenses if needed. If you are >> >> interested in getting a copy of MSDN, please contact me off-list. >> > >> > Does it include a license for Windows itself? >> > Does it allow me to install and run it in a VM? >> > If so, I'm interested. >> > >> Yes to both. MSDN offers a very broad license, with activation keys for >> many products generated on demand. >> > >The left panel on >http://msdn.microsoft.com/en-us/subscriptions/cc137115.aspx has the full >list of currently available products if anyone wants to know what's all >involved here. This is really awesome. I have an OEM license for Windows 7 but because I dual boot that on my primary Ubuntu development machine, I rarely use it. Being able to run Windows in a VM will mean I'll actually do it regularly. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From steve at holdenweb.com Mon Apr 19 17:31:52 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 19 Apr 2010 11:31:52 -0400 Subject: [Python-Dev] MSDN licenses available for python-dev In-Reply-To: <20100419104107.5d9bc674@heresy> References: <4BCC4316.4040207@holdenweb.com> <20100419104107.5d9bc674@heresy> Message-ID: Barry Warsaw wrote: > On Apr 19, 2010, at 08:14 AM, Brian Curtin wrote: > >> On Mon, Apr 19, 2010 at 06:48, Steve Holden wrote: >> >>> Antoine Pitrou wrote: >>>> Le Fri, 16 Apr 2010 08:01:54 -0500, Brian Curtin a ?crit : >>>>> The recent threads on builds/installers for Mac and Windows reminded me >>>>> of Steve Holden's push to get the python-dev team equipped via a >>>>> connection with the Microsoft Open Source Technology Center. The OSTC >>>>> team provides Microsoft Developer Network licenses to open source >>>>> projects to assist them in better supporting Windows. >>>>> >>>>> I've talked with Steve (who passed the task to me) and the Microsoft >>>>> folks, and they are happy to provide more licenses if needed. If you are >>>>> interested in getting a copy of MSDN, please contact me off-list. >>>> Does it include a license for Windows itself? >>>> Does it allow me to install and run it in a VM? >>>> If so, I'm interested. >>>> >>> Yes to both. MSDN offers a very broad license, with activation keys for >>> many products generated on demand. >>> >> The left panel on >> http://msdn.microsoft.com/en-us/subscriptions/cc137115.aspx has the full >> list of currently available products if anyone wants to know what's all >> involved here. > > This is really awesome. I have an OEM license for Windows 7 but because I > dual boot that on my primary Ubuntu development machine, I rarely use it. > Being able to run Windows in a VM will mean I'll actually do it regularly. > Well, the whole point from both sides is to ensure better support for Python on Microsoft platforms. So go to it! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From brett at python.org Mon Apr 19 20:24:59 2010 From: brett at python.org (Brett Cannon) Date: Mon, 19 Apr 2010 11:24:59 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <4BCB9EC7.9030309@gmx.de> <4BCBDA63.5080103@v.loewis.de> Message-ID: I'm with Martin; just let it be explicit and stick an example in the docs. -Brett (from his phone with hopefully only one copy sent of this email) On Apr 18, 2010 9:22 PM, Martin v. L?wis wrote: > - many optparse programs use the version argument > - many other programmers find this feature ver... You actually brought up a good reason yourself: In the face of ambiguity, refuse the temptation to guess. If you ask "give me a version argument", the question is "how is it spelled?". IIUC, you originally complained that the spelling of argparse (i.e. -v/--version) is not good, and that a different spelling should be used. So it's ambiguous, in which case the feature shouldn't be provided in the first place. Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev at python.org http:/... -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Mon Apr 19 20:50:15 2010 From: db3l.net at gmail.com (David Bolen) Date: Mon, 19 Apr 2010 14:50:15 -0400 Subject: [Python-Dev] MSDN licenses available for python-dev References: Message-ID: Antoine Pitrou writes: > Does it include a license for Windows itself? > Does it allow me to install and run it in a VM? > If so, I'm interested. Yes, in fact, it's due to the availability of this license that I was able to set up the Win7 buildbot. -- David From scott+python-dev at scottdial.com Mon Apr 19 21:19:22 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 19 Apr 2010 15:19:22 -0400 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCBB596.9020806@holdenweb.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> <4BCBB596.9020806@holdenweb.com> Message-ID: <4BCCACBA.8000303@scottdial.com> On 4/18/2010 9:44 PM, Steve Holden wrote: > Tobias Herp wrote: >> Steven Bethard schrieb: >>> On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: >>>> Steven Bethard schrieb: >>>>> But I'd really like a consensus about the correct behavior, and so far >>>>> I have not seen that. >> >> Do you take your own poll seriously? >> > When was this ever a democracy? Is consensus superficial? -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From fuzzyman at voidspace.org.uk Mon Apr 19 21:51:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 19 Apr 2010 21:51:00 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCCACBA.8000303@scottdial.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> <4BCBB596.9020806@holdenweb.com> <4BCCACBA.8000303@scottdial.com> Message-ID: <4BCCB424.2020604@voidspace.org.uk> On 19/04/2010 21:19, Scott Dial wrote: > On 4/18/2010 9:44 PM, Steve Holden wrote: > >> Tobias Herp wrote: >> >>> Steven Bethard schrieb: >>> >>>> On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: >>>> >>>>> Steven Bethard schrieb: >>>>> >>>>>> But I'd really like a consensus about the correct behavior, and so far >>>>>> I have not seen that. >>>>>> >>> Do you take your own poll seriously? >>> >>> >> When was this ever a democracy? >> > Is consensus superficial? > > No, but it isn't always possible or necessary. In general the maintainer of a module should make the best decision, not the one with the most backing. :-) All the best, Michael -- http://www.ironpythoninaction.com/ From martin at v.loewis.de Mon Apr 19 22:53:36 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Apr 2010 22:53:36 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: References: <201004191133.58602.victor.stinner@haypocalc.com> Message-ID: <4BCCC2D0.4070507@v.loewis.de> Antoine Pitrou wrote: > Victor Stinner haypocalc.com> writes: >> It's a choice, I didn't want to patch Windows because I know that Windows use >> unicode internally. I consider that developers using Python3 should use >> unicode on Windows, and byte or unicode+surrogates on other OS. > > I think both possibilities should be available on all OSes, so as to make it > easier to write cross-platform code. Having to switch being bytes and unicode > depending on the OS means developers will have to deal with encoding issues > themselves, which is suboptimal from a language usability's point of view. Indeed, you shouldn't be switching. Instead, you should be using Unicode strings all the time. Regards, Martin From martin at v.loewis.de Mon Apr 19 22:55:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Apr 2010 22:55:39 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: <201004191133.58602.victor.stinner@haypocalc.com> References: <201004191133.58602.victor.stinner@haypocalc.com> Message-ID: <4BCCC34B.8040000@v.loewis.de> > I'm working on surrogates in filenames on Linux (more generally on BSD and > UNIX OS) to support undecodable filenames, see PEP 383. Amaury told me that I > only fixed the non-Windows versions (I fixed subprocess about the current > directory and _ctypes.dlopen()), but it doesn't work on Windows. > > It's a choice, I didn't want to patch Windows because I know that Windows use > unicode internally. I consider that developers using Python3 should use > unicode on Windows, and byte or unicode+surrogates on other OS. > > I don't know well Windows API, and so I would like your opinion about that ;-) Can you please elaborate what the specific issue is? I completely fail to see what byte strings have to do with surrogate codes. AFAICT, on Windows, you can just use surrogate codes at the APIs, and be done. Regards, Martin From steven.bethard at gmail.com Tue Apr 20 00:12:06 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 19 Apr 2010 15:12:06 -0700 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCCB424.2020604@voidspace.org.uk> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCBABA8.4080704@gmx.de> <4BCBB596.9020806@holdenweb.com> <4BCCACBA.8000303@scottdial.com> <4BCCB424.2020604@voidspace.org.uk> Message-ID: On Mon, Apr 19, 2010 at 12:51 PM, Michael Foord wrote: > On 19/04/2010 21:19, Scott Dial wrote: >> Is consensus superficial? > > No, but it isn't always possible or necessary. In general the maintainer of > a module should make the best decision, not the one with the most backing. > :-) Yep, that was my logic. With 42% vs. 47% I decided there really wasn't consensus, and I should make the decision that I felt was best for argparse. If it had been e.g., 20% vs 80%, I would have bent to the will of the masses regardless of what I thought. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From nyamatongwe at gmail.com Tue Apr 20 00:39:07 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Tue, 20 Apr 2010 08:39:07 +1000 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: <201004191133.58602.victor.stinner@haypocalc.com> References: <201004191133.58602.victor.stinner@haypocalc.com> Message-ID: Victor Stinner: > It's a choice, I didn't want to patch Windows because I know that Windows use > unicode internally. I consider that developers using Python3 should use > unicode on Windows, and byte or unicode+surrogates on other OS. The Win32 byte string APIs convert their inputs to Unicode and then run Unicode code. You don't get additional capabilities by calling the byte string APIs and should avoid them completely. Including an easy way to invoke them on Windows will just lead to failures. People may think that Unix code that uses the byte string APIs for better platform fidelity can just run this code on Windows and get equivalent benefits. They won't and instead will see an inverted form of the problems they are trying to avoid on Unix. If there is ever a reason to use a byte string API on Windows (and I can't think of any) then ctypes can be used to explicitly call the API desired. Neil From victor.stinner at haypocalc.com Tue Apr 20 01:35:57 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 20 Apr 2010 01:35:57 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: <4BCCC34B.8040000@v.loewis.de> References: <201004191133.58602.victor.stinner@haypocalc.com> <4BCCC34B.8040000@v.loewis.de> Message-ID: <201004200135.57093.victor.stinner@haypocalc.com> Le lundi 19 avril 2010 22:55:39, vous avez ?crit : > Can you please elaborate what the specific issue is? Amaury reopened my issue #8393 "subprocess: support undecodable current working directory on POSIX OS" because "It does not work on Windows" (bytes are rejected). > I completely fail to see what byte strings have to do with surrogate > codes. AFAICT, on Windows, you can just use surrogate codes at the APIs, > and be done. Before my patch, subprocess used PyArg_ParseTuple(args, "...z...", ...) to parse the current working directory: surrogates were rejected. But I specified in my issue title that the issue is specific to "POSIX OS". I should replace it by "non-Windows". -- Amaury also reopened #8394 "ctypes.dlopen() doesn't support surrogates", because ctypes.CDLL() rejects byte string. On Windows, Python3 uses LoadLibraryW() to load a library, and the Python API rejects byte string. -- The question was: should we change python3 to accept byte strings on Windows? I think that I can re-close these two issues because it's a good thing to avoid the evil, locale dependent, mbcs encoding ;-) Unicode is a superset of mbcs. -- Victor Stinner http://www.haypocalc.com/ From phil at riverbankcomputing.com Tue Apr 20 14:19:44 2010 From: phil at riverbankcomputing.com (Phil Thompson) Date: Tue, 20 Apr 2010 13:19:44 +0100 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 Message-ID: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> When I build my C++ extension on Windows (specifically PyQt with MinGW) against Python v2.6.5 it fails to run under v2.6.4. The same problem exists when building against v3.1.2 and running under v3.1.1. The error message is... ImportError: DLL load failed: The specified procedure could not be found. ...though I don't know what the procedure is. When built against v2.6.4 it runs fine under all v2.6.x. When built under v3.1.1 it runs fine under all v3.1.x. I had always assumed that an extension built with vX.Y.Z would always run under vX.Y.Z-1. Am I wrong in that assumption, or is this a bug in the latest versions? Thanks, Phil From mail at timgolden.me.uk Tue Apr 20 14:38:28 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 20 Apr 2010 13:38:28 +0100 Subject: [Python-Dev] Planning a Windows buildbot Message-ID: <4BCDA044.3000703@timgolden.me.uk> I've got agreement in principle for one of our decommissioned servers to be repurposed as a Python buildbot. My idea would be to set it up as a Windows 2003 R2 server, at least partly because we don't seem to have any Windows server buildbots and it's a platform I'm especially interested in. Does anyone suggest that a different configuration would be preferable? Thanks to the PSF MSDN license I can pretty much choose whatever I want. I believe Brian Curtin's looking at the possibility of a Win2k8 box which is one reason why I steered away from that. Also because my box -- whenever they hand it over -- will be relatively old. TJG From cournape at gmail.com Tue Apr 20 14:50:51 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Apr 2010 21:50:51 +0900 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> Message-ID: On Tue, Apr 20, 2010 at 9:19 PM, Phil Thompson wrote: > When I build my C++ extension on Windows (specifically PyQt with MinGW) > against Python v2.6.5 it fails to run under v2.6.4. The same problem exists > when building against v3.1.2 and running under v3.1.1. > > The error message is... > > ImportError: DLL load failed: The specified procedure could not be found. > > ...though I don't know what the procedure is. > > When built against v2.6.4 it runs fine under all v2.6.x. When built under > v3.1.1 it runs fine under all v3.1.x. > > I had always assumed that an extension built with vX.Y.Z would always run > under vX.Y.Z-1. I don't know how well it is handled in python, but this is extremely hard to do in general - you are asking about forward compatibility, not backward compatibility. Is there a reason why you need to do this ? The usual practice is to build against the *oldest* compatible version you can, so that it remains compatible with everything afterwards, cheers, David From phil at riverbankcomputing.com Tue Apr 20 15:07:44 2010 From: phil at riverbankcomputing.com (Phil Thompson) Date: Tue, 20 Apr 2010 14:07:44 +0100 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> Message-ID: <8632c13ba441da2f0cc4048c8d8ba881@localhost> On Tue, 20 Apr 2010 21:50:51 +0900, David Cournapeau wrote: > On Tue, Apr 20, 2010 at 9:19 PM, Phil Thompson > wrote: >> When I build my C++ extension on Windows (specifically PyQt with MinGW) >> against Python v2.6.5 it fails to run under v2.6.4. The same problem >> exists >> when building against v3.1.2 and running under v3.1.1. >> >> The error message is... >> >> ImportError: DLL load failed: The specified procedure could not be found. >> >> ...though I don't know what the procedure is. >> >> When built against v2.6.4 it runs fine under all v2.6.x. When built under >> v3.1.1 it runs fine under all v3.1.x. >> >> I had always assumed that an extension built with vX.Y.Z would always run >> under vX.Y.Z-1. > > I don't know how well it is handled in python, but this is extremely > hard to do in general - you are asking about forward compatibility, > not backward compatibility. Yes, I know. > Is there a reason why you need to do this ? The usual practice is to > build against the *oldest* compatible version you can, so that it > remains compatible with everything afterwards, I'm happy to do that if that is the recommendation, although this is the first time I've noticed this sort of problem (since v1.5). Phil From tds333+pydev at gmail.com Tue Apr 20 15:31:51 2010 From: tds333+pydev at gmail.com (Wolfgang Langner) Date: Tue, 20 Apr 2010 15:31:51 +0200 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> Message-ID: <4BCDACC7.20404@gmail.com> Hi, On 20.04.2010 14:19, Phil Thompson wrote: > When I build my C++ extension on Windows (specifically PyQt with MinGW) > against Python v2.6.5 it fails to run under v2.6.4. The same problem exists > when building against v3.1.2 and running under v3.1.1. > > The error message is... > > ImportError: DLL load failed: The specified procedure could not be found. > > ...though I don't know what the procedure is. > > When built against v2.6.4 it runs fine under all v2.6.x. When built under > v3.1.1 it runs fine under all v3.1.x. > > I had always assumed that an extension built with vX.Y.Z would always run > under vX.Y.Z-1. > > Am I wrong in that assumption, or is this a bug in the latest versions? It is also possible, that this is a windows issue with Manifest and DLL loading. Welcome in the manifest hell. If Version 2.6.4 is build against another version of the msvcrt as version 2.6.5 than this could happen. Possible solution is to not automatically build the manifest and use one with hard coded versions across one Python version 2.6.X. Regards Wolfgang From ndbecker2 at gmail.com Tue Apr 20 20:55:28 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 20 Apr 2010 14:55:28 -0400 Subject: [Python-Dev] argparse ambiguity handling Message-ID: I've noticed argparse ambiguity handling has changed a bit over last few revisions. I have cases where 1 valid input is a prefix of another: e.g.: '--string' '--string2' With the most recent 1.1, the behavior is: --string=hello is accepted, while: --strin=hello is marked as ambiguous. I'm OK with this, but just want to know if there is agreement that this is the behavior we want. From steven.bethard at gmail.com Tue Apr 20 21:10:00 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 20 Apr 2010 12:10:00 -0700 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: References: Message-ID: On Tue, Apr 20, 2010 at 11:55 AM, Neal Becker wrote: > I've noticed argparse ambiguity handling has changed a bit over last few > revisions. > > I have cases where 1 valid input is a prefix of another: > > e.g.: > '--string' > '--string2' > > With the most recent 1.1, the behavior is: > > --string=hello > > is accepted, while: > > --strin=hello > > is marked as ambiguous. > > I'm OK with this, but just want to know if there is agreement that this is > the behavior we want. I don't have a strong feeling about this. What was the behavior before? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ndbecker2 at gmail.com Tue Apr 20 21:27:53 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 20 Apr 2010 15:27:53 -0400 Subject: [Python-Dev] argparse ambiguity handling References: Message-ID: Steven Bethard wrote: > On Tue, Apr 20, 2010 at 11:55 AM, Neal Becker wrote: >> I've noticed argparse ambiguity handling has changed a bit over last few >> revisions. >> >> I have cases where 1 valid input is a prefix of another: >> >> e.g.: >> '--string' >> '--string2' >> >> With the most recent 1.1, the behavior is: >> >> --string=hello >> >> is accepted, while: >> >> --strin=hello >> >> is marked as ambiguous. >> >> I'm OK with this, but just want to know if there is agreement that this >> is the behavior we want. > > I don't have a strong feeling about this. What was the behavior before? > > Steve At least 1 earlier version said that even exact match was ambiguous. I have a preference to allow at least exact matches to succeed even in the case of ambiguity - mainly because I accidentally created this already once, and I feel it's better to at least work somewhat. Not sure if there is any more elegant solution. OTOH, I feel this is somewhat inelegant, as it appears to treat exact match as a special case. From brett at python.org Tue Apr 20 21:42:24 2010 From: brett at python.org (Brett Cannon) Date: Tue, 20 Apr 2010 12:42:24 -0700 Subject: [Python-Dev] Email addresses for new committers for python-committers Message-ID: If you are a committer and are NOT subscribed to the python-committers mailing list (I believe this at least includes Giampaolo, JP, and Brian), then please either reply to this email with your preferred email address or let me know directly (the former is preferred so Georg or Eric can beat me to the subscription if I take too long). -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Apr 20 21:48:15 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 20 Apr 2010 15:48:15 -0400 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: References: Message-ID: <20100420194814.075273A40F5@sparrow.telecommunity.com> At 03:27 PM 4/20/2010 -0400, Neal Becker wrote: >I have a preference to allow at least exact matches to succeed even in the >case of ambiguity - mainly because I accidentally created this already once, >and I feel it's better to at least work somewhat. Not sure if there is any >more elegant solution. OTOH, I feel this is somewhat inelegant, as it >appears to treat exact match as a special case. How about throwing an error the moment you define a set of options that create this potential ambiguity? ;-) (i.e. force you to define --string1 and --string2 instead of --string and --string2) (Either that, or have an option to turn off ambiguous guessing, and throw the error unless you've turned it off.) From brian.curtin at gmail.com Tue Apr 20 21:49:32 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 20 Apr 2010 14:49:32 -0500 Subject: [Python-Dev] Email addresses for new committers for python-committers In-Reply-To: References: Message-ID: On Tue, Apr 20, 2010 at 14:42, Brett Cannon wrote: > If you are a committer and are NOT subscribed to the python-committers > mailing list (I believe this at least includes Giampaolo, JP, and Brian), > then please either reply to this email with your preferred email address or > let me know directly (the former is preferred so Georg or Eric can beat me > to the subscription if I take too long). > I'm already subscribed via my first name dot last name @gmail.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon+python-dev at unequivocal.co.uk Tue Apr 20 21:54:32 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Tue, 20 Apr 2010 20:54:32 +0100 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: References: Message-ID: <20100420195432.GF31281@snowy.squish.net> On Tue, Apr 20, 2010 at 03:27:53PM -0400, Neal Becker wrote: > I have a preference to allow at least exact matches to succeed even in the > case of ambiguity - mainly because I accidentally created this already once, > and I feel it's better to at least work somewhat. Not sure if there is any > more elegant solution. OTOH, I feel this is somewhat inelegant, as it > appears to treat exact match as a special case. Surely an exact match *is* a special case - it's an exact match! From martin at v.loewis.de Tue Apr 20 22:19:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Apr 2010 22:19:34 +0200 Subject: [Python-Dev] Support byte string API of Windows in Python3? In-Reply-To: <201004200135.57093.victor.stinner@haypocalc.com> References: <201004191133.58602.victor.stinner@haypocalc.com> <4BCCC34B.8040000@v.loewis.de> <201004200135.57093.victor.stinner@haypocalc.com> Message-ID: <4BCE0C56.8080706@v.loewis.de> > Amaury reopened my issue #8393 "subprocess: support undecodable current > working directory on POSIX OS" because "It does not work on Windows" (bytes > are rejected). I see. I'd like to know whether that's an incompatible change. We shouldn't make incompatible changes in that matter. However, if you were not able to pass a bytes cwd on Windows before, it's fine that you still are not able to. I'm puzzled how your patch could have possibly affected Windows, since it was only change _posixsubprocess.c. So I'm closing the report again. > Amaury also reopened #8394 "ctypes.dlopen() doesn't support surrogates", > because ctypes.CDLL() rejects byte string. Ok, I'll close this as well. Regards, Martin From martin at v.loewis.de Tue Apr 20 22:24:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Apr 2010 22:24:44 +0200 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> Message-ID: <4BCE0D8C.4070907@v.loewis.de> Phil Thompson wrote: > When I build my C++ extension on Windows (specifically PyQt with MinGW) > against Python v2.6.5 it fails to run under v2.6.4. The same problem exists > when building against v3.1.2 and running under v3.1.1. > > The error message is... > > ImportError: DLL load failed: The specified procedure could not be found. > > ...though I don't know what the procedure is. > > When built against v2.6.4 it runs fine under all v2.6.x. When built under > v3.1.1 it runs fine under all v3.1.x. > > I had always assumed that an extension built with vX.Y.Z would always run > under vX.Y.Z-1. > > Am I wrong in that assumption, or is this a bug in the latest versions? You are not wrong in that assumption, but it still might not be a bug in the latest version. It could also be a bug in MingW or PyQt. Before we can judge on that, we need to understand what exactly happened. As a starting point for further research, try the sxstrace utility of your Vista installation. Regards, Martin From martin at v.loewis.de Tue Apr 20 22:39:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Apr 2010 22:39:11 +0200 Subject: [Python-Dev] Planning a Windows buildbot In-Reply-To: <4BCDA044.3000703@timgolden.me.uk> References: <4BCDA044.3000703@timgolden.me.uk> Message-ID: <4BCE10EF.7060009@v.loewis.de> Tim Golden wrote: > I've got agreement in principle for one of our decommissioned > servers to be repurposed as a Python buildbot. My idea would > be to set it up as a Windows 2003 R2 server, at least partly > because we don't seem to have any Windows server buildbots > and it's a platform I'm especially interested in. > > Does anyone suggest that a different configuration would be > preferable? Thanks to the PSF MSDN license I can pretty much > choose whatever I want. I believe Brian Curtin's looking at > the possibility of a Win2k8 box which is one reason why I > steered away from that. Also because my box -- whenever they > hand it over -- will be relatively old. If you are truly adventurous (sp?), you could try to setup a Cygwin or MingW target - I don't think we have one of these yet. However, doing so may be pointless as it may be doomed to fail, anyway, in which case there is no point in continuously verifying that failure. If you are a little adventurous, set up VS 2010. We don't have any build slaves for that, either, plus one would have to define a way how the slaves convert the project files (which could either happen manually in SVN, as originally done for VS 2008, or automatically through VS, or automatically through something like PCbuild/vs9to8.py). As for the operating system: I don't think it matters at all. Windows XP Home is as fine a platform for this application as Windows 2008 R2 Enterprise. What would matter is processor architecture - we don't have an Itanium Windows buildbot slave :-) Regards, Martin From mail at timgolden.me.uk Tue Apr 20 22:59:45 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 20 Apr 2010 21:59:45 +0100 Subject: [Python-Dev] Email addresses for new committers for python-committers In-Reply-To: References: Message-ID: <4BCE15C1.4050406@timgolden.me.uk> On 20/04/2010 20:42, Brett Cannon wrote: > If you are a committer and are NOT subscribed to the python-committers > mailing list (I believe this at least includes Giampaolo, JP, and Brian), > then please either reply to this email with your preferred email address or > let me know directly (the former is preferred so Georg or Eric can beat me > to the subscription if I take too long). I did subscribe earlier today, and replied to the usual "please confirm" email, but instead of the conventional "You're on the list" email I got a slightly ambiguous "You're due to be moderated" email. If you can confirm that I'm subscribed then all well and good. If not, let me know if there's something else I should do. TJG From solipsis at pitrou.net Tue Apr 20 23:01:36 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 20 Apr 2010 21:01:36 +0000 (UTC) Subject: [Python-Dev] Planning a Windows buildbot References: <4BCDA044.3000703@timgolden.me.uk> <4BCE10EF.7060009@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > As for the operating system: I don't think it matters at all. Windows XP > Home is as fine a platform for this application as Windows 2008 R2 > Enterprise. What would matter is processor architecture - we don't have > an Itanium Windows buildbot slave Microsoft has said it would stop doing Windows releases for Itanium: http://arstechnica.com/microsoft/news/2010/04/microsoft-its-the-end-of-the-line-for-itanium-support.ars I'm not sure there's any point in caring about it anymore. Regards Antoine. From martin at v.loewis.de Tue Apr 20 23:09:55 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 20 Apr 2010 23:09:55 +0200 Subject: [Python-Dev] Planning a Windows buildbot In-Reply-To: References: <4BCDA044.3000703@timgolden.me.uk> <4BCE10EF.7060009@v.loewis.de> Message-ID: <4BCE1823.3090601@v.loewis.de> Antoine Pitrou wrote: > Martin v. L?wis v.loewis.de> writes: >> As for the operating system: I don't think it matters at all. Windows XP >> Home is as fine a platform for this application as Windows 2008 R2 >> Enterprise. What would matter is processor architecture - we don't have >> an Itanium Windows buildbot slave > > Microsoft has said it would stop doing Windows releases for Itanium: > http://arstechnica.com/microsoft/news/2010/04/microsoft-its-the-end-of-the-line-for-itanium-support.ars > > I'm not sure there's any point in caring about it anymore. That's why I put a smiley in that sentence. Regards, Martin From phil at riverbankcomputing.com Tue Apr 20 23:17:56 2010 From: phil at riverbankcomputing.com (Phil Thompson) Date: Tue, 20 Apr 2010 22:17:56 +0100 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: <4BCE0D8C.4070907@v.loewis.de> References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> <4BCE0D8C.4070907@v.loewis.de> Message-ID: <4bb20fc395b1d9e9c0ce43c45340fb3a@localhost> On Tue, 20 Apr 2010 22:24:44 +0200, "Martin v. L?wis" wrote: > Phil Thompson wrote: >> When I build my C++ extension on Windows (specifically PyQt with MinGW) >> against Python v2.6.5 it fails to run under v2.6.4. The same problem >> exists >> when building against v3.1.2 and running under v3.1.1. >> >> The error message is... >> >> ImportError: DLL load failed: The specified procedure could not be found. >> >> ...though I don't know what the procedure is. >> >> When built against v2.6.4 it runs fine under all v2.6.x. When built under >> v3.1.1 it runs fine under all v3.1.x. >> >> I had always assumed that an extension built with vX.Y.Z would always run >> under vX.Y.Z-1. >> >> Am I wrong in that assumption, or is this a bug in the latest versions? > > You are not wrong in that assumption, but it still might not be a bug in > the latest version. It could also be a bug in MingW or PyQt. > > Before we can judge on that, we need to understand what exactly happened. > > As a starting point for further research, try the sxstrace utility of > your Vista installation. What Vista installation? XP I'm afraid... Phil From Tobias.Herp at gmx.de Tue Apr 20 23:22:16 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Tue, 20 Apr 2010 23:22:16 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <20100418221946.39d6160c@heresy> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <20100418221946.39d6160c@heresy> Message-ID: <4BCE1B08.50505@gmx.de> Barry Warsaw schrieb: > On Apr 18, 2010, at 07:30 PM, Eric Smith wrote: > >> Steven Bethard wrote: >>> By the way, we could simplify the typical add_argument usage by adding >>> "show program's version number and exit" as the default help for the >>> 'version' action. Then you should just write: >>> >>> parser.add_argument('--version', action='version', version='') >> >> I like this the best. I don't like argparse adding arguments for me. > > I concur. This gives me all the flexibility I need to make my programs accept > exactly the arguments I want and print exactly the information I want to > present. But you had this possibility all the time! No deprecation or removal of the simple and convenient 'version' argument is needed for this. Just omit it, and build your version action yourself. But please notice that there are others who appreciate this simple way to do it and don't need more. -- Tobias From martin at v.loewis.de Tue Apr 20 23:59:43 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 20 Apr 2010 23:59:43 +0200 Subject: [Python-Dev] Binary Compatibility Issue with Python v2.6.5 and v3.1.2 In-Reply-To: <4bb20fc395b1d9e9c0ce43c45340fb3a@localhost> References: <47a2d9a794e464b0f8e1f8b45333ee7c@localhost> <4BCE0D8C.4070907@v.loewis.de> <4bb20fc395b1d9e9c0ce43c45340fb3a@localhost> Message-ID: <4BCE23CF.6020606@v.loewis.de> >> As a starting point for further research, try the sxstrace utility of >> your Vista installation. > > What Vista installation? XP I'm afraid... Then you are out of look. Check the event log, and ask in Windows help forums for better ways to analyse the problem. Regards, Martin From ncoghlan at gmail.com Wed Apr 21 00:16:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Apr 2010 08:16:23 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCE1B08.50505@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <20100418221946.39d6160c@heresy> <4BCE1B08.50505@gmx.de> Message-ID: <4BCE27B7.6030007@gmail.com> Tobias Herp wrote: > No deprecation or removal of the simple and convenient 'version' > argument is needed for this. Just omit it, and build your version > action yourself. But please notice that there are others who appreciate > this simple way to do it and don't need more. One Obvious Way. In this case, the ambiguity of the "version=" argument is sufficient to elevate it to the level of "attractive nuisance" (particularly given the legacy behaviour of conflicting with typical verbosity options). The default help for the action needs to be fixed, and there are a couple of documentation tweaks to be made, but Steven's decision to remove the feature has a sound rationale. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Wed Apr 21 00:31:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Apr 2010 00:31:14 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCE27B7.6030007@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <20100418221946.39d6160c@heresy> <4BCE1B08.50505@gmx.de> <4BCE27B7.6030007@gmail.com> Message-ID: <4BCE2B32.8070806@v.loewis.de> Nick Coghlan wrote: > Tobias Herp wrote: >> No deprecation or removal of the simple and convenient 'version' >> argument is needed for this. Just omit it, and build your version >> action yourself. But please notice that there are others who appreciate >> this simple way to do it and don't need more. > > One Obvious Way. > > In this case, the ambiguity of the "version=" argument is sufficient to > elevate it to the level of "attractive nuisance" (particularly given the > legacy behaviour of conflicting with typical verbosity options). The > default help for the action needs to be fixed, and there are a couple of > documentation tweaks to be made, but Steven's decision to remove the > feature has a sound rationale. Exactly my feelings. IMO, straight removal would be fine as well, but a deprecation may be more user-friendly, wrt. the existing code base. Regards, Martin From Tobias.Herp at gmx.de Wed Apr 21 00:34:10 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Wed, 21 Apr 2010 00:34:10 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCE27B7.6030007@gmail.com> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <20100418221946.39d6160c@heresy> <4BCE1B08.50505@gmx.de> <4BCE27B7.6030007@gmail.com> Message-ID: <4BCE2BE2.40100@gmx.de> Nick Coghlan schrieb: > Tobias Herp wrote: >> No deprecation or removal of the simple and convenient 'version' >> argument is needed for this. Just omit it, and build your version >> action yourself. But please notice that there are others who appreciate >> this simple way to do it and don't need more. > > One Obvious Way. One /Obvious/ Way! To simply check off the "give the program a version" task, /this/ (the version argument) is the way to do it (obvious enough for legions of programmers who are able to use optparse). If you need a more fancy version action, of course, you need to take care of that yourself. > In this case, the ambiguity of the "version=" argument ... What /is/ this ambiguity of the "version=" argument? There is none! Do it the way optparse does it, and all is fine! -- Tobias From Tobias.Herp at gmx.de Wed Apr 21 01:46:14 2010 From: Tobias.Herp at gmx.de (Tobias Herp) Date: Wed, 21 Apr 2010 01:46:14 +0200 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCBDA63.5080103@v.loewis.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <4BCB9EC7.9030309@gmx.de> <4BCBDA63.5080103@v.loewis.de> Message-ID: <4BCE3CC6.4040506@gmx.de> "Martin v. L?wis" schrieb: >> - many optparse programs use the version argument >> - many other programmers find this feature very convenient >> - dropping or deprecating this is a totally unnecessary change >> (I have not read a single real reason /why/ this should be done). > > You actually brought up a good reason yourself: > > In the face of ambiguity, refuse the temptation to guess. I didn't guess. The vast majority of *x commandline tools, *including the Python interpreter*, uses '--version' and/or '-V' for the version action, while '-v' is often used for something else, most often verbosity (again: e.g. the Python interpreter). Nobody has argued the converse. The optparse module uses just '--version' option string by default, following the standard. > If you ask "give me a version argument", the question is "how is it > spelled?". IIUC, you originally complained that the spelling of argparse > (i.e. -v/--version) is not good, and that a different spelling should be > used. So it's ambiguous, in which case the feature shouldn't be provided > in the first place. Wrong! Just follow the de-facto standard. There has never been a problem with optparse's version argument, simply because of the fact it conforms with the standard (using just '--version'). Perhaps we wouldn't argue now if they had decided to add '-V' as well, which is AFAIK /never/ used for anything else than the version, either; perhaps this would have been adopted by argparse as well. The optik/optparse people didn't, and the argparse people apparently didn't really look out what options look like in the wild. There is *no* problem with a 'version' argument (which has always been *optional*; nobody is forced to use it) which simply defines the '--version' option to trigger the version option. There has *never* been a problem with the optparse way to do it! Now argparse wants to become a part of Python's standard library; it certainly has advantages over optparse. But the user won't be interested in the difference: "Oh, I see, this is an argparse-based program, so the '-v' is naturally not available for verbosity. Fine!" Up to today, the argparse module had a limited audience. When becoming part of Python, it must follow standards; it should Do The Right Thing by default, and it should keep the usage difference to optparse as small as possible. People should be allowed to rely on the fact that /Python scripts follow the standards/; this is a huge usability gain. Thus, argparse should *continue* to support the version argument, because it is convenient (we use Python because it is convenient!), and it is sufficient in 95% of the cases (I have written lots of optparse scripts, and in exactly one case I wrote the version output myself). It should discontinue the '-v' short option string, because it is non-standard, and it is ambiguous; it is often used for something else, while '-V' isn't (but adding this as a default would like break existing argparse programs, so this is not an option). If the community feels this is necessary, the '-v' could be supported 'invisibly': it should not be visible in the help, and it must not block the usage of '-v' for other purposes. -- Tobias From cs at zip.com.au Wed Apr 21 04:11:22 2010 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 21 Apr 2010 12:11:22 +1000 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: References: Message-ID: <20100421021122.GA2916@cskk.homeip.net> On 20Apr2010 15:27, Neal Becker wrote: | Steven Bethard wrote: | | > On Tue, Apr 20, 2010 at 11:55 AM, Neal Becker wrote: | >> I've noticed argparse ambiguity handling has changed a bit over last few | >> revisions. | >> | >> I have cases where 1 valid input is a prefix of another: | >> | >> e.g.: | >> '--string' | >> '--string2' | >> | >> With the most recent 1.1, the behavior is: | >> | >> --string=hello | >> | >> is accepted, while: | >> | >> --strin=hello | >> | >> is marked as ambiguous. | >> | >> I'm OK with this, but just want to know if there is agreement that this | >> is the behavior we want. | > | > I don't have a strong feeling about this. What was the behavior before? | > Steve | | At least 1 earlier version said that even exact match was ambiguous. | | I have a preference to allow at least exact matches to succeed even in the | case of ambiguity - mainly because I accidentally created this already once, | and I feel it's better to at least work somewhat. Not sure if there is any | more elegant solution. OTOH, I feel this is somewhat inelegant, as it | appears to treat exact match as a special case. I think the new behaviour is desirable. Plenty of commands have both --foo and --foo-tweak. Usually because --foo came first and --foo-tweak came later, or simply because --foo is the simple and obvious and commonly wanted mode and --foo-tweak is a special case. Real world example: rsync and the --delete* options. I'm sure plenty of others can be found. The new behaviour makes this doable. The old behaviour made it unimplementable. Maybe it is desirable to be _able_ to forbid this arguably-ambiguous option set, but I definitely feel that the current behaviour should be available. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ If it wasn't for lawyers, we wouldn't need them. From stephen at xemacs.org Wed Apr 21 05:41:27 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 21 Apr 2010 12:41:27 +0900 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCE3CC6.4040506@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <4BCB9EC7.9030309@gmx.de> <4BCBDA63.5080103@v.loewis.de> <4BCE3CC6.4040506@gmx.de> Message-ID: <871ve9mpzs.fsf@uwakimon.sk.tsukuba.ac.jp> Tobias Herp writes: > (we use Python because it is convenient!), Speak for yourself, I want no part of that "we". I use Python because it's well defined and well documented and makes sense and provides powerful operations and runs quickly enough in those applications where I use it. The fact that it's not excessively inconvenient despite an avowed bias toward EIBTI is the icing, not the cake. Remember, convenience is explicitly subsidiary: "not every 3 line function needs to be a builtin." I have no particular opinion the rest of the debate. From regebro at gmail.com Wed Apr 21 06:56:39 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 21 Apr 2010 06:56:39 +0200 Subject: [Python-Dev] Doctests REPORT_ONLY_FIRST_FAILURE ignored Message-ID: If you set up any sort of report flag on the unit test itself, the default report flags given to the testrunner are ignored. This goes for all report flags, the REPORT_xDIFF and REPORT_ONLY_FIRST_FAILURE. I'd suggest that we do allow the testrunner to set REPORT_ONLY_FIRST_FAILURE by default even if the testcase has a REPORT_xDIFF flag. Why? Because it makes sense. :) The REPORT_xDFF flags makes sense both as parameters to a testrunner, and as flags on a testcase. You might want to permanently set diff flags on tests that generate output that warrants a diff if they fail. REPORT_ONLY_FIRST_FAILURE you would rarely set on a testcase. You don't want that on the testcase, as buildbots wouldn't see the subsequent fails, and developers might think it was only a minor issue. It is a flag you give to the testrunner to stop having the first failure scroll off screen. So you want it to work *always*. You don't want or expect it to stop working just because one testcase had a DIFF flag set. Right? Or did I miss something? If not, I'll provide a patch and put it in the bugtracker. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From eric at trueblade.com Wed Apr 21 09:53:16 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 21 Apr 2010 03:53:16 -0400 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: <20100421021122.GA2916@cskk.homeip.net> References: <20100421021122.GA2916@cskk.homeip.net> Message-ID: <4BCEAEEC.8030207@trueblade.com> Cameron Simpson wrote: > On 20Apr2010 15:27, Neal Becker wrote: > | Steven Bethard wrote: > | > | > On Tue, Apr 20, 2010 at 11:55 AM, Neal Becker wrote: > | >> I've noticed argparse ambiguity handling has changed a bit over last few > | >> revisions. > | >> > | >> I have cases where 1 valid input is a prefix of another: > | >> > | >> e.g.: > | >> '--string' > | >> '--string2' > | >> > | >> With the most recent 1.1, the behavior is: > | >> > | >> --string=hello > | >> > | >> is accepted, while: > | >> > | >> --strin=hello > | >> > | >> is marked as ambiguous. > | >> > | >> I'm OK with this, but just want to know if there is agreement that this > | >> is the behavior we want. > | > > | > I don't have a strong feeling about this. What was the behavior before? > | > Steve > | > | At least 1 earlier version said that even exact match was ambiguous. > | > | I have a preference to allow at least exact matches to succeed even in the > | case of ambiguity - mainly because I accidentally created this already once, > | and I feel it's better to at least work somewhat. Not sure if there is any > | more elegant solution. OTOH, I feel this is somewhat inelegant, as it > | appears to treat exact match as a special case. > > I think the new behaviour is desirable. Plenty of commands have both > --foo and --foo-tweak. Usually because --foo came first and --foo-tweak > came later, or simply because --foo is the simple and obvious and > commonly wanted mode and --foo-tweak is a special case. > > Real world example: rsync and the --delete* options. I'm sure plenty of > others can be found. > > The new behaviour makes this doable. The old behaviour made it > unimplementable. Maybe it is desirable to be _able_ to forbid this > arguably-ambiguous option set, but I definitely feel that the current > behaviour should be available. I agree the new behavior is desirable. And I also think it should be the default, although I feel less strongly about that. But since this behavior seems to be an accident of the implementation (based on Steve's comment above), I think a test should be added for it if it's accepted as a Good Thing (tm). Otherwise it's possible that it will vanish as the implementation changes. Eric. From jon+python-dev at unequivocal.co.uk Wed Apr 21 11:14:39 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Wed, 21 Apr 2010 10:14:39 +0100 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: <4BCEAEEC.8030207@trueblade.com> References: <20100421021122.GA2916@cskk.homeip.net> <4BCEAEEC.8030207@trueblade.com> Message-ID: <20100421091439.GK31281@snowy.squish.net> On Wed, Apr 21, 2010 at 03:53:16AM -0400, Eric Smith wrote: > I agree the new behavior is desirable. And I also think it should be the > default, although I feel less strongly about that. > > But since this behavior seems to be an accident of the implementation > (based on Steve's comment above), I think a test should be added for it > if it's accepted as a Good Thing (tm). Otherwise it's possible that it > will vanish as the implementation changes. +1 to all of that From barry at python.org Wed Apr 21 15:40:05 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 21 Apr 2010 09:40:05 -0400 Subject: [Python-Dev] PEP 328, relative imports and Python 2.7 Message-ID: <20100421094005.3adea983@heresy> While talking about Python 2.6 -> 2.7 transitions, the subject of relative and absolute imports has come up. PEP 328 states that absolute imports will be enabled by default in Python 2.7, however I cannot verify that this has actually happened. * Misc/NEWS doesn't mention it * What's New in Python 2.7 doesn't mention it either * My own limited testing on trunk indicates it hasn't happened Now, truth be told I only spent 15 minutes investigating this so I could certainly have missed something. Are absolute imports enabled by default in Python 2.7? If not, given that we're into beta, I don't think we can do it now, so I would suggest updating the PEP. If they have been enabled then we need to update NEWS and What's New. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Apr 21 15:48:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Apr 2010 23:48:48 +1000 Subject: [Python-Dev] Python 2.7b1 and argparse's version action In-Reply-To: <4BCE2BE2.40100@gmx.de> References: <4BCADF66.8070702@gmx.de> <4BCB7B0A.6000305@gmx.de> <4BCB95FF.8050202@trueblade.com> <20100418221946.39d6160c@heresy> <4BCE1B08.50505@gmx.de> <4BCE27B7.6030007@gmail.com> <4BCE2BE2.40100@gmx.de> Message-ID: <4BCF0240.8060407@gmail.com> Tobias Herp wrote: > > What /is/ this ambiguity of the "version=" argument? There is none! > > Do it the way optparse does it, and all is fine! All is not fine. How do I add a "-V" shortcut that is equivalent to "--version" as part of an optparse program? (That's a real question, I honestly have no idea how I would do that short of writing my own version display option). Since argparse now uses action='version' instead, the same question becomes trivial (just list "-V" as one of the accepted options for that action). There are at least three reasonable approaches for version options (ignoring the fact that Windows users expect their options to start with a '/' instead of a '-'): - just "--version" (the optparse way) - "-V" and "--version" (the Python interpreter way) - "-v" and "--version" (the old argparse way) The third is by far the least common of the three, but even you admit both of the first two are quite common and acceptable according to the de facto "standard". Optparse makes it easy to do it the optparse way, but difficult to support either of the other formats. Argparse used to be in a similar boat, where only the third option was easy. The latest argparse approach is straightforward regardless of the particular spelling you happen to prefer. Since getting away from the "thou shalt" preachiness of optparse is one of the big reasons for adding argparse in the first place, it seems odd to complain when it switches to a more consistent and flexible API. There were a couple of issues to be addressed with the implementation and the documentation, but the basic approach is sound. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dickinsm at gmail.com Wed Apr 21 15:56:22 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 21 Apr 2010 14:56:22 +0100 Subject: [Python-Dev] PEP 328, relative imports and Python 2.7 In-Reply-To: <20100421094005.3adea983@heresy> References: <20100421094005.3adea983@heresy> Message-ID: On Wed, Apr 21, 2010 at 2:40 PM, Barry Warsaw wrote: > While talking about Python 2.6 -> 2.7 transitions, the subject of relative and > absolute imports has come up. ?PEP 328 states that absolute imports will be > enabled by default in Python 2.7, however I cannot verify that this has > actually happened. I'm fairly sure it hasn't. I brought this up on python-dev in February (around Feb 2nd; thread entitled 'Absolute imports in Python 2.x'), but for some reason I can only find the tail end of that thread on mail.python.org: http://mail.python.org/pipermail/python-dev/2010-February/097458.html > Python 2.7? ?If not, given that we're into beta, I don't think we can do it > now, so I would suggest updating the PEP. Agreed. There's also the question of whether deprecation warnings or -3 warnings should be raised; see http://bugs.python.org/issue7844 Mark From dickinsm at gmail.com Wed Apr 21 15:58:54 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 21 Apr 2010 14:58:54 +0100 Subject: [Python-Dev] PEP 328, relative imports and Python 2.7 In-Reply-To: References: <20100421094005.3adea983@heresy> Message-ID: On Wed, Apr 21, 2010 at 2:56 PM, Mark Dickinson wrote: > On Wed, Apr 21, 2010 at 2:40 PM, Barry Warsaw wrote: >> While talking about Python 2.6 -> 2.7 transitions, the subject of relative and >> absolute imports has come up. ?PEP 328 states that absolute imports will be >> enabled by default in Python 2.7, however I cannot verify that this has >> actually happened. > > I'm fairly sure it hasn't. ?I brought this up on python-dev in > February (around Feb 2nd; ?thread entitled 'Absolute imports in Python > 2.x'), but for some reason I can only find the tail end of that thread > on mail.python.org: > > http://mail.python.org/pipermail/python-dev/2010-February/097458.html Ah, here's a better link to a different archive of the previous discussion. http://www.mail-archive.com/python-dev at python.org/msg45275.html Mark From hancock.robert at gmail.com Wed Apr 21 16:04:40 2010 From: hancock.robert at gmail.com (Robert Hancock) Date: Wed, 21 Apr 2010 10:04:40 -0400 Subject: [Python-Dev] jean-claude elias Message-ID: http://www.aricciafestaecultura.it/home.php From g.rodola at gmail.com Wed Apr 21 16:38:22 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Wed, 21 Apr 2010 16:38:22 +0200 Subject: [Python-Dev] Email addresses for new committers for python-committers In-Reply-To: References: Message-ID: I'm already subscribed as "g.rodola at gmail.com" and I'm able to receive messages from the list. 2010/4/20 Brett Cannon : > If you are a committer and are NOT subscribed to the python-committers > mailing list (I believe this at least includes Giampaolo, JP, and Brian), > then please either reply to this email with your preferred email address or > let me know directly (the former is preferred so Georg or Eric can beat me > to the subscription if I take too long). > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com > > From rob.cliffe at btinternet.com Wed Apr 21 18:17:55 2010 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 21 Apr 2010 17:17:55 +0100 Subject: [Python-Dev] Small suggestion re help(Exception) Message-ID: help() on an Exception class lists the method resolution order (effectively the inheritance hierarchy). E.g. help(ArithmeticError) displays inter alia: Method resolution order: ArithmeticError StandardError Exception BaseException __builtin__.object Would it be possible for it also display the Python built-in SUBclasses of the class? E.g. in the same example something like: Built-in subclasses: FloatingPointError OverflowError ZeroDivisionError This may seem pointless to grizzled old Python veterans who (maybe) know the inheritance hierarchy backwards, but for those of us with less experience I think it would be helpful, e.g. (1) It would help to track down an Exception whose name you have forgotten. (2) It would sometimes be illuminating, e.g. LookupError might be a bit obscure at first, until you see that it is the superclass of the familiar IndexError and KeyError. Sorry, I'm not sure if I should send this sort of thing to Python-Ideas or Python-Dev, so please let me know gently which one I should (not) have sent it to. Best wishes Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Wed Apr 21 18:22:36 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 21 Apr 2010 18:22:36 +0200 Subject: [Python-Dev] Small suggestion re help(Exception) In-Reply-To: References: Message-ID: <4BCF264C.90909@voidspace.org.uk> On 21/04/2010 18:17, Rob Cliffe wrote: > help() on an Exception class lists the method resolution order > (effectively the inheritance hierarchy). > E.g. help(ArithmeticError) displays inter alia: > Method resolution order: > ArithmeticError > StandardError > Exception > BaseException > __builtin__.object > Would it be possible for it also display the Python built-in > SUBclasses of the class? E.g. in the same example something like: > Built-in subclasses: > FloatingPointError > OverflowError > ZeroDivisionError > This may seem pointless to grizzled old Python veterans who (maybe) > know the inheritance hierarchy backwards, but for those of us with > less experience I think it would be helpful, e.g. > (1) It would help to track down an Exception whose name you have > forgotten. > (2) It would sometimes be illuminating, e.g. LookupError might be a > bit obscure at first, until you see that it is the superclass of the > familiar IndexError and KeyError. > Sorry, I'm not sure if I should send this sort of thing to > Python-Ideas or Python-Dev, so please let me know gently which one I > should (not) have sent it to. This sounds like a pretty straightforward feature request that you could put on the tracker (preferably with patch and tests of course), but is on-topic for this mailing list. I rarely use help(...) myself (I probably should), but it sounds like a useful enhancement. Michael > Best wishes > Rob Cliffe > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.bethard at gmail.com Wed Apr 21 18:33:32 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 21 Apr 2010 09:33:32 -0700 Subject: [Python-Dev] argparse ambiguity handling In-Reply-To: <20100421091439.GK31281@snowy.squish.net> References: <20100421021122.GA2916@cskk.homeip.net> <4BCEAEEC.8030207@trueblade.com> <20100421091439.GK31281@snowy.squish.net> Message-ID: On Wed, Apr 21, 2010 at 2:14 AM, Jon Ribbens wrote: > On Wed, Apr 21, 2010 at 03:53:16AM -0400, Eric Smith wrote: >> I agree the new behavior is desirable. And I also think it should be the >> default, although I feel less strongly about that. >> >> But since this behavior seems to be an accident of the implementation >> (based on Steve's comment above), I think a test should be added for it >> if it's accepted as a Good Thing (tm). Otherwise it's possible that it >> will vanish as the implementation changes. > > +1 to all of that Turns out I actually fixed this on purpose in r79 of argparse, and even added a test (TestOptionalsDoubleDashPrefixMatch) to test this behavior. So also +1 from me as well since I've already done it. ;-) Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From rob.cliffe at btinternet.com Wed Apr 21 18:52:08 2010 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 21 Apr 2010 17:52:08 +0100 Subject: [Python-Dev] Small suggestion re help(Exception) Message-ID: <3F49C95140FF4CEB87023B4EC9295C74@robslaptop> ----- Original Message ----- From: "Aur?lien Campeas" To: "Rob Cliffe" Sent: Wednesday, April 21, 2010 5:25 PM Subject: Re: [Python-Dev] Small suggestion re help(Exception) > like in > >>>> ArithmeticError.__subclasses__() > [, 'exceptions.OverflowError'>, ] > > ? Yes, exactly like that. I wasn't aware of __subclasses__ (it doesn't appear in dir(ArithmeticError), for example), but if my suggestion were adopted, __subclasses__ would be just what is needed to implement it. Rob Cliffe > > Le mercredi 21 avril 2010 ? 17:17 +0100, Rob Cliffe a ?crit : >> help() on an Exception class lists the method resolution order >> (effectively the inheritance hierarchy). >> E.g. help(ArithmeticError) displays inter alia: >> >> Method resolution order: >> ArithmeticError >> StandardError >> Exception >> BaseException >> __builtin__.object >> >> Would it be possible for it also display the Python built-in >> SUBclasses of the class? E.g. in the same example something like: >> >> Built-in subclasses: >> FloatingPointError >> OverflowError >> ZeroDivisionError >> >> This may seem pointless to grizzled old Python veterans who (maybe) >> know the inheritance hierarchy backwards, but for those of us with >> less experience I think it would be helpful, e.g. >> (1) It would help to track down an Exception whose name you have >> forgotten. >> (2) It would sometimes be illuminating, e.g. LookupError might be a >> bit obscure at first, until you see that it is the superclass of the >> familiar IndexError and KeyError. >> >> Sorry, I'm not sure if I should send this sort of thing to >> Python-Ideas or Python-Dev, so please let me know gently which one I >> should (not) have sent it to. >> >> Best wishes >> Rob Cliffe >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/aurelien.campeas%40logilab.fr > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Apr 21 19:47:56 2010 From: brett at python.org (Brett Cannon) Date: Wed, 21 Apr 2010 10:47:56 -0700 Subject: [Python-Dev] PEP 328, relative imports and Python 2.7 In-Reply-To: References: <20100421094005.3adea983@heresy> Message-ID: On Wed, Apr 21, 2010 at 06:58, Mark Dickinson wrote: > On Wed, Apr 21, 2010 at 2:56 PM, Mark Dickinson > wrote: > > On Wed, Apr 21, 2010 at 2:40 PM, Barry Warsaw wrote: > >> While talking about Python 2.6 -> 2.7 transitions, the subject of > relative and > >> absolute imports has come up. PEP 328 states that absolute imports will > be > >> enabled by default in Python 2.7, however I cannot verify that this has > >> actually happened. > > > > I'm fairly sure it hasn't. I brought this up on python-dev in > > February (around Feb 2nd; thread entitled 'Absolute imports in Python > > 2.x'), but for some reason I can only find the tail end of that thread > > on mail.python.org: > > > > http://mail.python.org/pipermail/python-dev/2010-February/097458.html > > Ah, here's a better link to a different archive of the previous discussion. > > http://www.mail-archive.com/python-dev at python.org/msg45275.html And it looks like it mostly got hijacked by a discussion of how to keep track of long-term issues. =) As for changing the semantics of this, I would love to see it happen, but as Barry said, into a beta probably means no. So we should at least add a Py3K warning (if there isn't one already) as implicit relative imports cannot be guaranteed correct by 2to3 in the face of ambiguity. Plus the silencing of DeprecationWarnings by default means it won't trigger more noise for users. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Apr 21 22:05:33 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 21 Apr 2010 16:05:33 -0400 Subject: [Python-Dev] PEP 328, relative imports and Python 2.7 In-Reply-To: References: <20100421094005.3adea983@heresy> Message-ID: <20100421160533.2d085a96@heresy> On Apr 21, 2010, at 02:56 PM, Mark Dickinson wrote: >On Wed, Apr 21, 2010 at 2:40 PM, Barry Warsaw wrote: >> While talking about Python 2.6 -> 2.7 transitions, the subject of relative and >> absolute imports has come up. ?PEP 328 states that absolute imports will be >> enabled by default in Python 2.7, however I cannot verify that this has >> actually happened. > >I'm fairly sure it hasn't. I brought this up on python-dev in >February (around Feb 2nd; thread entitled 'Absolute imports in Python >2.x'), but for some reason I can only find the tail end of that thread >on mail.python.org: > >http://mail.python.org/pipermail/python-dev/2010-February/097458.html > >> Python 2.7? ?If not, given that we're into beta, I don't think we can do it >> now, so I would suggest updating the PEP. > >Agreed. There's also the question of whether deprecation warnings or >-3 warnings should be raised; see > >http://bugs.python.org/issue7844 For the time being, I've removed the sentence from the PEP that says absolute imports will be enabled by default in Python 2.7. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From victor.stinner at haypocalc.com Wed Apr 21 22:16:54 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 21 Apr 2010 22:16:54 +0200 Subject: [Python-Dev] Reject bytearray filename in Python 3.2 Message-ID: <201004212216.54138.victor.stinner@haypocalc.com> Hi, About my work on unicode surrogates, I would like to reject bytearray filename, especially in the PyUnicode_FSConverter function. As explained in issue #8485, support bytearray requires to test the result type, acquire/release a lock, the API is more complex, etc. I don't know real world usecase of bytearray filename. The os.path module doesn't support it and no Python 3.0 or 3.1 user noticed that. Support of bytearray in not documentation in PyUnicode_FSConverter() comment (Include/unicodeobject.h). Martin Loewis, the author of PEP 383, created PyUnicode_FSConverter() and he agree to drop support of bytearray filename. But he asked me to ask on this mailing list if anyone would prefer to mark bytearray filename as deprecated in 3.2 and reject them in 3.3. I will be very sad if someone ask me to keep bytearray filename support in 3.2 because I opened a lot of issues about surrogates and I would make my work more diffcult :-( -- Victor Stinner http://www.haypocalc.com/ From ncoghlan at gmail.com Thu Apr 22 00:18:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Apr 2010 08:18:50 +1000 Subject: [Python-Dev] Small suggestion re help(Exception) In-Reply-To: <3F49C95140FF4CEB87023B4EC9295C74@robslaptop> References: <3F49C95140FF4CEB87023B4EC9295C74@robslaptop> Message-ID: <4BCF79CA.9090106@gmail.com> Rob Cliffe wrote: > Yes, exactly like that. > I wasn't aware of __subclasses__ (it doesn't appear in > dir(ArithmeticError), > for example) Slight tangent: dir() usually includes class attributes (e.g. "set(dir(1)) >= set(dir(int))"), but to prevent weirdness that particular feature is omitted for the metaclass of a type. So dir(type(cls)) may reveal a few extra methods/properties (such as mro() and __subclasses__()) that aren't listed when doing dir(cls). > but if my suggestion were adopted, __subclasses__ would be > just what is needed to implement it. Restricting it to just builtins would be tricky, especially for Exception (even without importing any additional libraries, Exception.__subclasses__() lists 23 direct subclasses at the interpreter prompt for me). Anyway, as Michael suggested, a feature request on the bugs.python.org tracker is the best way forward. That's no guarantee that anyone will pick it up and run with it, but at least it will record the idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Apr 22 00:21:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Apr 2010 08:21:02 +1000 Subject: [Python-Dev] Reject bytearray filename in Python 3.2 In-Reply-To: <201004212216.54138.victor.stinner@haypocalc.com> References: <201004212216.54138.victor.stinner@haypocalc.com> Message-ID: <4BCF7A4E.5080202@gmail.com> Victor Stinner wrote: > I will be very sad if someone ask me to keep bytearray filename support in 3.2 > because I opened a lot of issues about surrogates and I would make my work > more diffcult :-( I don't have an opinion one way or the other regarding bytearray, but even if you deprecated it rather than dropping it, couldn't you just add the surrogate support for the Unicode path and leave the bytecode path with the legacy behaviour? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From victor.stinner at haypocalc.com Thu Apr 22 01:05:36 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 22 Apr 2010 01:05:36 +0200 Subject: [Python-Dev] Reject bytearray filename in Python 3.2 In-Reply-To: <201004212216.54138.victor.stinner@haypocalc.com> References: <201004212216.54138.victor.stinner@haypocalc.com> Message-ID: <201004220105.36532.victor.stinner@haypocalc.com> Le jeudi 22 avril 2010 00:21:02, vous avez ?crit : > Victor Stinner wrote: > > I will be very sad if someone ask me to keep bytearray filename support > > in 3.2 because I opened a lot of issues about surrogates and I would make > > my work more diffcult :-( > > I don't have an opinion one way or the other regarding bytearray, but > even if you deprecated it rather than dropping it, couldn't you just add > the surrogate support for the Unicode path and leave the bytecode path > with the legacy behaviour? Yes, we can do everything. But does it really have a sense? No Python function using filenames return a bytearray object. Example: os.listdir() and os.walk() result type is bytes or str. Changing PyUnicode_FSConverter() is trivial, but the problem is in the function calling PyUnicode_FSConverter(). The caller have to support byte and bytearray. Antoine proposed to convert bytearray to bytes to ensure that PyUnicode_FSConverter() result is a bytes object. But in this case, we still need also to fix ntpath, ,posixpath and macpath modules to support bytearray. -- Victor Stinner http://www.haypocalc.com/ From ziade.tarek at gmail.com Thu Apr 22 10:54:08 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 22 Apr 2010 10:54:08 +0200 Subject: [Python-Dev] PEP 376 Message-ID: Hello, I think I went through all the latest feedback regarding PEP 376. There will be still some work of course, on the implementation side (for instance the Zip issues described by PJE). But I would like to go ahead and propose PEP 376 for acceptance. If it's accepted, I will implement it in pkgutil for Python 3.2 and provide a backport in distutils2. (In fact Michael Mulich has started this work in distutils2, so it might be done the other way) The next big piece is the FHS-compatible handling of resource files, which will worth a PEP on its own. Regards Tarek -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Thu Apr 22 13:32:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Apr 2010 21:32:54 +1000 Subject: [Python-Dev] Reject bytearray filename in Python 3.2 In-Reply-To: <201004220105.36532.victor.stinner@haypocalc.com> References: <201004212216.54138.victor.stinner@haypocalc.com> <201004220105.36532.victor.stinner@haypocalc.com> Message-ID: <4BD033E6.9090706@gmail.com> Victor Stinner wrote: > Le jeudi 22 avril 2010 00:21:02, vous avez ?crit : >> Victor Stinner wrote: >>> I will be very sad if someone ask me to keep bytearray filename support >>> in 3.2 because I opened a lot of issues about surrogates and I would make >>> my work more diffcult :-( >> I don't have an opinion one way or the other regarding bytearray, but >> even if you deprecated it rather than dropping it, couldn't you just add >> the surrogate support for the Unicode path and leave the bytecode path >> with the legacy behaviour? > > Yes, we can do everything. But does it really have a sense? No Python function > using filenames return a bytearray object. Example: os.listdir() and os.walk() > result type is bytes or str. Oh, never mind then, I misunderstood the question ('bytearray' flipped to 'bytes' in my brain). I don't see the point in allowing a mutable argument either. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From victor.stinner at haypocalc.com Thu Apr 22 13:39:38 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 22 Apr 2010 13:39:38 +0200 Subject: [Python-Dev] PyTuple_Parse*() and discontigious buffer Message-ID: <201004221339.38421.victor.stinner@haypocalc.com> Hi, I opened issue #8215 about PyTuple_Parse*() outdated documentation. When I read the code, I found two functions: getbuffer() and convertbuffer(). getbuffer() checks that the buffer is contigious, whereas convertbuffer() doesn't. We can group formats by contigious check: - contigious buffer: s, y*, z*, w* - discontigious buffer: s#, y, z, t, w, w# I tried to test formats accepting discontigious buffer but I didn't found any function or library to test this. I'm not sure that functions using s#, y, z, t, w, w# format do really support discontigious buffer and it's just a bug in getargs.c. Do someone have information about that? Should it be fixed (only accept contigious buffer)? -- Victor Stinner http://www.haypocalc.com/ From asmodai at in-nomine.org Thu Apr 22 13:58:31 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu, 22 Apr 2010 13:58:31 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: References: Message-ID: <20100422115831.GC1219@nexus.in-nomine.org> -On [20100422 10:55], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >The next big piece is the FHS-compatible handling of resource files, >which will worth a PEP on its own. You do realize, I hope, that FHS is only followed by Linux distributions and not even fully at that. BSD Unixes, for example, have quite some different hier(7) requirements, most notably that /usr/local is used. So is your intent to have a PEP solely for Linux systems? -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B A wise man that walks in the dark with a blindfold on, is not much of a wise man... From eric at trueblade.com Thu Apr 22 14:02:06 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 22 Apr 2010 08:02:06 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <20100422115831.GC1219@nexus.in-nomine.org> References: <20100422115831.GC1219@nexus.in-nomine.org> Message-ID: <4BD03ABE.3030806@trueblade.com> Jeroen Ruigrok van der Werven wrote: > -On [20100422 10:55], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >> The next big piece is the FHS-compatible handling of resource files, >> which will worth a PEP on its own. > > You do realize, I hope, that FHS is only followed by Linux distributions and > not even fully at that. BSD Unixes, for example, have quite some different > hier(7) requirements, most notably that /usr/local is used. > > So is your intent to have a PEP solely for Linux systems? > I believe the intent is to have enough metadata describing the resource files that any system could be supported, including FHS. -- Eric. From ziade.tarek at gmail.com Thu Apr 22 14:16:08 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 22 Apr 2010 14:16:08 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4BD03ABE.3030806@trueblade.com> References: <20100422115831.GC1219@nexus.in-nomine.org> <4BD03ABE.3030806@trueblade.com> Message-ID: 2010/4/22 Eric Smith : > Jeroen Ruigrok van der Werven wrote: >> >> -On [20100422 10:55], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >>> >>> The next big piece is the FHS-compatible handling of resource files, >>> which will worth a PEP on its own. >> >> You do realize, I hope, that FHS is only followed by Linux distributions >> and >> not even fully at that. BSD Unixes, for example, have quite some different >> hier(7) requirements, most notably that /usr/local is used. >> >> So is your intent to have a PEP solely for Linux systems? >> > > I believe the intent is to have enough metadata describing the resource > files that any system could be supported, including FHS. Yes that's the idea. We want to provide a way in Python, to describe for each system (Windows included) default paths for resource files, and to give the ability to system packagers to change them. At the end, the developer will be able to say "Foo is a help file", and leave it up to distutils to install it at the best place depending on the platform. Then, he'll be able to say "Let me have Foo" at runtime. And os packagers, for example in Ubuntu, will be able to say "for now on, help files will be located *here*". We did quite a lot of work already on this during Pycon, see : http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst FHS was just our insipiration to list the different targets, because it's one of the most advanced description in this field. But we do need to provide default locations under windows as well, and windows will possibly add new locations, who knows.. Regards Tarek -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Thu Apr 22 17:42:06 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 22 Apr 2010 17:42:06 +0200 Subject: [Python-Dev] Doctests REPORT_ONLY_FIRST_FAILURE ignored In-Reply-To: References: Message-ID: On Wed, Apr 21, 2010 at 06:56, Lennart Regebro wrote: > If you set up any sort of report flag on the unit test itself, the > default report flags given to the testrunner are ignored. This goes > for all report flags, the REPORT_xDIFF and REPORT_ONLY_FIRST_FAILURE. > I'd suggest that we do allow the testrunner to set > REPORT_ONLY_FIRST_FAILURE by default even if the testcase has a > REPORT_xDIFF flag. Why? Because it makes sense. :) > > The REPORT_xDFF flags makes sense both as parameters to a testrunner, > and as flags on a testcase. You might want to permanently set diff > flags on tests that generate output that warrants a diff if they fail. > > REPORT_ONLY_FIRST_FAILURE you would rarely set on a testcase. You > don't want that on the testcase, as buildbots wouldn't see the > subsequent fails, and developers might think it was only a minor > issue. It is a flag you give to the testrunner to stop having the > first failure scroll off screen. So you want it to work *always*. You > don't want or expect it to stop working just because one testcase had > a DIFF flag set. Right? > > Or did I miss something? If not, I'll provide a patch and put it in > the bugtracker. I apparently didn't miss anything. :) Patch will arrive soon. :) From pje at telecommunity.com Thu Apr 22 18:02:41 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 22 Apr 2010 12:02:41 -0400 Subject: [Python-Dev] PEP 376 Message-ID: <20100422160240.D2B443A4070@sparrow.telecommunity.com> At 10:54 AM 4/22/2010 +0200, Tarek Ziad? wrote: >Hello, > >I think I went through all the latest feedback regarding PEP 376. > >There will be still some work of course, on the implementation side >(for instance the Zip issues described by PJE). > >But I would like to go ahead and propose PEP 376 for acceptance. One final (optional) question/suggestion... should we change from this (in RECORD files): lib/python2.6/site-packages/docutils/__init__.pyc to this: lib/python2.6/site-packages/docutils/__init__.pyc,, In this way, reader code can be written as: for path, hash, size in csv.reader(...): It's not a high-priority thing, but if anybody is writing code to parse RECORD files outside the provided API implementation (eg. system packaging tool authors), they'll probably appreciate this. ;-) From ziade.tarek at gmail.com Thu Apr 22 18:05:32 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 22 Apr 2010 18:05:32 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <20100422160154.803693A4070@sparrow.telecommunity.com> References: <20100422160154.803693A4070@sparrow.telecommunity.com> Message-ID: 2010/4/22 P.J. Eby : > At 10:54 AM 4/22/2010 +0200, Tarek Ziad? wrote: >> >> Hello, >> >> I think I went through all the latest feedback regarding PEP 376. >> >> There will be still some work of course, on the implementation side >> (for instance the Zip issues described by PJE). >> >> But I would like to go ahead and propose PEP 376 for acceptance. > > One final (optional) question/suggestion... ?should we change from this (in > RECORD files): > > ? ?lib/python2.6/site-packages/docutils/__init__.pyc > > to this: > > ? ?lib/python2.6/site-packages/docutils/__init__.pyc,, > > In this way, reader code can be written as: > > ? ?for path, hash, size in csv.reader(...): > > It's not a high-priority thing, but if anybody is writing code to parse > RECORD files outside the provided API implementation (eg. system packaging > tool authors), they'll probably appreciate this. ?;-) > Yes, of course. the RECORD file is supposed to be iterable using the csv reader, so that's a mistake in the PEP. Thanks for noticing, I'll fix it later today > -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Thu Apr 22 22:14:48 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 22 Apr 2010 13:14:48 -0700 Subject: [Python-Dev] code.python.org - random 403 errors Message-ID: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> I am seeing random 403 errors when cloning the mercurial repositories of Python. $ hg clone http://code.python.org/hg/branches/release2.6-maint destination directory: release2.6-maint requesting all changes abort: HTTP Error 403: Forbidden $ hg clone http://code.python.org/hg/branches/release2.6-maint/ destination directory: release2.6-maint requesting all changes abort: HTTP Error 403: Forbidden Sometimes they work, sometimes not. Should I report this issue to a specific tracker (other than bugs.python.org and webmaster at python.org)? -srid From martin at v.loewis.de Thu Apr 22 22:45:38 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Apr 2010 22:45:38 +0200 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> Message-ID: <4BD0B572.80904@v.loewis.de> Sridhar Ratnakumar wrote: > I am seeing random 403 errors when cloning the mercurial repositories of Python. > > $ hg clone http://code.python.org/hg/branches/release2.6-maint > destination directory: release2.6-maint > requesting all changes > abort: HTTP Error 403: Forbidden > $ hg clone http://code.python.org/hg/branches/release2.6-maint/ > destination directory: release2.6-maint > requesting all changes > abort: HTTP Error 403: Forbidden > > Sometimes they work, sometimes not. What's your IP address? (you may want to tell in private only). > Should I report this issue to a specific tracker (other than bugs.python.org and webmaster at python.org)? Please don't report that to the tracker. Regards, Martin From stefan at bytereef.org Thu Apr 22 23:03:51 2010 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 22 Apr 2010 23:03:51 +0200 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> Message-ID: <20100422210351.GA16581@yoda.bytereef.org> Sridhar Ratnakumar wrote: > I am seeing random 403 errors when cloning the mercurial repositories of Python. > > $ hg clone http://code.python.org/hg/branches/release2.6-maint > destination directory: release2.6-maint > requesting all changes > abort: HTTP Error 403: Forbidden > $ hg clone http://code.python.org/hg/branches/release2.6-maint/ > destination directory: release2.6-maint > requesting all changes > abort: HTTP Error 403: Forbidden > > Sometimes they work, sometimes not. In general python.org seems flaky today. Sometimes I could not reach the site at all, now a ping to svn.python.org gives 30% packet loss. Stefan Krah From martin at v.loewis.de Fri Apr 23 00:01:17 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Apr 2010 00:01:17 +0200 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <20100422210351.GA16581@yoda.bytereef.org> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> <20100422210351.GA16581@yoda.bytereef.org> Message-ID: <4BD0C72D.6090609@v.loewis.de> > In general python.org seems flaky today. Sometimes I could not reach the site > at all, now a ping to svn.python.org gives 30% packet loss. Yes. See http://www.python.org/munin/localdomain/localhost.localdomain-if_eth0.html Somebody was consuming all bandwidth, although we couldn't quite figure out who (whenever I was looking, the traffic looked genuine). It turned out that Sridhar himself contributed a good chunk of this traffic. Regards, Martin From sridharr at activestate.com Fri Apr 23 02:45:50 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 22 Apr 2010 17:45:50 -0700 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <4BD0C72D.6090609@v.loewis.de> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> <20100422210351.GA16581@yoda.bytereef.org> <4BD0C72D.6090609@v.loewis.de> Message-ID: <3578E3A2-2CBD-4737-BAB1-EFF4B0894F80@activestate.com> On 2010-04-22, at 3:01 PM, Martin v. L?wis wrote: >> In general python.org seems flaky today. Sometimes I could not reach the site >> at all, now a ping to svn.python.org gives 30% packet loss. > > Yes. See > > http://www.python.org/munin/localdomain/localhost.localdomain-if_eth0.html > > Somebody was consuming all bandwidth, although we couldn't quite figure > out who (whenever I was looking, the traffic looked genuine). It turned > out that Sridhar himself contributed a good chunk of this traffic. Background in private emails to Martin (below): On 2010-04-22, at 5:42 PM, Sridhar Ratnakumar wrote: > > On 2010-04-22, at 2:13 PM, Martin v. L?wis wrote: > >>> [snip ... ip address] >>> >>> I must note that we do a nightly build of ActivePython using Python >>> mercurial repositories for trunk, release26-maint and release31-maint >>> (3 URLs) on 5 of our build machines. So one nightly build will make >>> 3x5=15 "hg clone" requests per day to code.python.org. This week, I >>> have been debugging the nightly build code (run by hudson); so that >>> number is likely higher this week. >> >> [snip ... possibly security-confidential part] >> >> Can you please find a way not to clone the complete repository every >> time? IIUC, it should be possible to update a clone, rather than >> refetching it from scratch. > > Ok, I setup a cron job to maintain an internal mirror of the above mentioned repositories in code.python.org. We'll do a "hg pull -u" (equivalent to "svn up") every hour; no clones. Hopefully, that should reduce the amount of requests from our side. Let me know if in future this issue repeats. From asmodai at in-nomine.org Fri Apr 23 07:55:21 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Fri, 23 Apr 2010 07:55:21 +0200 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <3578E3A2-2CBD-4737-BAB1-EFF4B0894F80@activestate.com> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> <20100422210351.GA16581@yoda.bytereef.org> <4BD0C72D.6090609@v.loewis.de> <3578E3A2-2CBD-4737-BAB1-EFF4B0894F80@activestate.com> Message-ID: <20100423055521.GC3225@nexus.in-nomine.org> -On [20100423 02:48], Sridhar Ratnakumar (sridharr at activestate.com) wrote: >> Ok, I setup a cron job to maintain an internal mirror of the above mentioned repositories in code.python.org. We'll do a "hg pull -u" (equivalent to "svn up") every hour; no clones. Hopefully, that should reduce the amount of requests from our side. Let me know if in future this issue repeats. Dirk Jan can probably correct me (or some other heavy Hg user) but for all I know you should indeed simply clone once and subsequently hg pull and from your local copy clone as you like. (At least that's also how http://wiki.services.openoffice.org/wiki/Mercurial/Getting_Started seems to aim at explaining.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B A wise man that walks in the dark with a blindfold on, is not much of a wise man... From jobh at broadpark.no Fri Apr 23 15:41:55 2010 From: jobh at broadpark.no (Joachim B Haga) Date: Fri, 23 Apr 2010 15:41:55 +0200 Subject: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals()) Message-ID: <85sk6mffq4.fsf@jobh.org> There seem to be an inconsistency in the handling of local scopes in exec. Consider the following code, which raises NameError if the '#' is removed from the second last line. block = """ b = 'ok' def f(): print(b) # raises NameError here f() """ scope = locals()#.copy() exec(block, globals(), scope) The intermediate scope is searched for the variable name if the third argument to exec() is locals(), but not if it is locals().copy(). Testing further, it looks like NameError is raised for any dict which is not identically equal to either globals() or locals(). This behaviour is quite unexpected, and I believe it qualifies as a bug. Tested with python 2.6.5 and 3.1.2. -- Joachim B Haga From doko at ubuntu.com Fri Apr 23 17:26:59 2010 From: doko at ubuntu.com (Matthias Klose) Date: Fri, 23 Apr 2010 17:26:59 +0200 Subject: [Python-Dev] autoconf update to 2.65 and cleanup Message-ID: <4BD1BC43.5020707@ubuntu.com> configure is still generated by 2.61; would it be possible to update to 2.65? The cr_lf issue mentioned in [1] seems to be resolved, ac_cr is now defined as ac_cr=`echo X | tr X '\015'` Proposing to - fix some quoting in help strings and code snippets (#8509) - update to autoconf 2.65 (#8510) - convert obsolete macros (AC_HELP_STRING, AC_TRY_*, AC_AIX, ...) one by one (tracking these in separate reports). Could this be done for both trunk and py3k branch, even if 2.7 already is in beta? Matthias [1] http://mail.python.org/pipermail/python-dev/2008-November/083781.html From victor.stinner at haypocalc.com Fri Apr 23 17:44:20 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 23 Apr 2010 17:44:20 +0200 Subject: [Python-Dev] autoconf update to 2.65 and cleanup In-Reply-To: <4BD1BC43.5020707@ubuntu.com> References: <4BD1BC43.5020707@ubuntu.com> Message-ID: <201004231744.21040.victor.stinner@haypocalc.com> Le vendredi 23 avril 2010 17:26:59, Matthias Klose a ?crit : > configure is still generated by 2.61; would it be possible to update to > 2.65? Yes, everything is possible. Open a new issue and write a patch ;-) > even if 2.7 already is in beta? I'm not sure that it's a good idea to change the build process after the first beta. It would depend on the issue comments ;-) -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Fri Apr 23 18:07:22 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 23 Apr 2010 18:07:22 +0200 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> Message-ID: <201004231807.22189.victor.stinner@haypocalc.com> Le jeudi 22 avril 2010 22:14:48, Sridhar Ratnakumar a ?crit : > I am seeing random 403 errors when cloning the mercurial repositories of > Python. I don't know if it is related, but I get errors from the bbreport tool: -------------- $ python2.6 bbreport.py 3.x Selected builders: 20 / 80 (branch: 3.x) Traceback (most recent call last): File "bbreport.py", line 903, in builders = main() File "bbreport.py", line 853, in main for xrb in proxy.getLastBuildsAllBuilders(limit): File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) xmlrpclib.Fault: -------------- Did the buildbot configuration/version changed recently? -- Victor Stinner http://www.haypocalc.com/ From status at bugs.python.org Fri Apr 23 18:09:03 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 23 Apr 2010 18:09:03 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100423160903.252D07812B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-04-16 - 2010-04-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2665 open (+58) / 17664 closed (+31) / 20329 total (+89) Open issues with patches: 1084 Average duration of open issues: 725 days. Median duration of open issues: 494 days. Open Issues Breakdown open 2621 (+58) languishing 9 ( +0) pending 34 ( +0) Issues Created Or Reopened (94) _______________________________ Obsolete RFCs should be removed from doc of urllib.urlparse 2010-04-19 CLOSED http://bugs.python.org/issue5650 reopened ezio.melotti patch, easy [PATCH] Drop "Computer" from "Apple Computer" in plistlib 2010-04-20 http://bugs.python.org/issue7852 reopened haypo patch configure: ignore AC_PROG_CC hardcoded CFLAGS 2010-04-23 http://bugs.python.org/issue8211 reopened lemburg patch subprocess: support undecodable current working directory on P 2010-04-19 CLOSED http://bugs.python.org/issue8393 reopened amaury.forgeotdarc patch ctypes.dlopen() doesn't support surrogates 2010-04-19 CLOSED http://bugs.python.org/issue8394 reopened amaury.forgeotdarc patch tiger buildbot: unable to resolv hostname address 2010-04-16 CLOSED http://bugs.python.org/issue8421 created haypo tiger buildbot: test_abspath_issue3426 failure (test_genericpa 2010-04-16 CLOSED http://bugs.python.org/issue8422 created haypo buildbot tiger buildbot: test_pep277 failures 2010-04-16 http://bugs.python.org/issue8423 created haypo buildbot Test assumptions for test_itimer_virtual and test_itimer_prof 2010-04-16 http://bugs.python.org/issue8424 created haypo patch, buildbot a -= b should be fast if a is a small set and b is a large set 2010-04-16 http://bugs.python.org/issue8425 created abacabadabacaba easy multiprocessing.Queue fails to get() very large objects 2010-04-16 http://bugs.python.org/issue8426 created Ian.Davis toplevel jumps to another location on the screen 2010-04-16 http://bugs.python.org/issue8427 created aparasch buildbot: test_multiprocessing timeout (test_notify_all? test_ 2010-04-16 http://bugs.python.org/issue8428 created haypo buildbot buildbot: test_subprocess timeout 2010-04-16 http://bugs.python.org/issue8429 created haypo buildbot test_site failure with non-ASCII directory 2010-04-17 CLOSED http://bugs.python.org/issue8430 created haypo patch, buildbot buildbot: hung on ARM Debian 2010-04-17 http://bugs.python.org/issue8431 created haypo buildbot build: test_send_signal of test_subprocess failure 2010-04-17 http://bugs.python.org/issue8432 created haypo buildbot buildbot: test_curses failure, getmouse() returned ERR 2010-04-17 http://bugs.python.org/issue8433 created haypo buildbot buildbot: test_gdb failure on sparc Ubuntu trunk 2010-04-17 CLOSED http://bugs.python.org/issue8434 created haypo buildbot It is possible to observe a mutating frozenset 2010-04-17 CLOSED http://bugs.python.org/issue8435 created abacabadabacaba set.__init__ accepts keyword args 2010-04-17 CLOSED http://bugs.python.org/issue8436 created abacabadabacaba easy test_gdb: gdb.Frame has no attribute function 2010-04-17 CLOSED http://bugs.python.org/issue8437 created loewis patch Codecs: "surrogateescape" error handler in Python 2.7 2010-04-18 CLOSED http://bugs.python.org/issue8438 created ysj.ray patch test_linecache failing in py3k r80169 2010-04-18 CLOSED http://bugs.python.org/issue8439 created tim.golden patch test_heapq interfering with test_import on py3k 2010-04-18 http://bugs.python.org/issue8440 created tim.golden Framework build broken in 3.2 brunk 2010-04-18 CLOSED http://bugs.python.org/issue8441 created ronaldoussoren Broken zipfile with python 3.2 on osx 2010-04-18 http://bugs.python.org/issue8442 created ronaldoussoren Broken zipfile with python 3.2 on osx 2010-04-18 CLOSED http://bugs.python.org/issue8443 created ronaldoussoren openssl version detection doesn't work properly when using OSX 2010-04-18 http://bugs.python.org/issue8444 created ronaldoussoren buildbot: test_ttk_guionly failures (test_traversal, test_tab_ 2010-04-18 http://bugs.python.org/issue8445 created haypo buildbot buildbot: DeprecationWarning not raised for icglue (test_py3kw 2010-04-18 http://bugs.python.org/issue8446 created haypo buildbot buildbot: test_httpservers failure (No module named operator) 2010-04-18 http://bugs.python.org/issue8447 created haypo buildbot buildbot: test_subprocess failure (test_no_leaking, Broken pip 2010-04-18 http://bugs.python.org/issue8448 created haypo buildbot buildbot: test_dbm and test_dbm_ndbm failures 2010-04-18 http://bugs.python.org/issue8449 created haypo buildbot httplib: false BadStatusLine() raised 2010-04-18 http://bugs.python.org/issue8450 created mschu syslog.syslog('msg') logs message with ident "python". 2010-04-19 CLOSED http://bugs.python.org/issue8451 created jafo patch, easy, needs review build_installer.py breaks bzip compilation 2010-04-19 CLOSED http://bugs.python.org/issue8453 created loewis unittest Module Problem with different Kinds of Invocation 2010-04-19 http://bugs.python.org/issue8454 created AP buildbot: test_urllib2_localnet failures (Connection refused) 2010-04-19 http://bugs.python.org/issue8455 created haypo buildbot sqlite3.connect documentation is incorrect 2010-04-19 http://bugs.python.org/issue8456 created AndiDog buildbot: test_asynchat and test_smtplib failures on Tiger: [E 2010-04-19 http://bugs.python.org/issue8457 created haypo buildbot: test_cmd_line failure on Tiger: [Errno 9] Bad file d 2010-04-19 http://bugs.python.org/issue8458 created haypo buildbot buildbot: test_select failure (test_returned_list_identity) on 2010-04-19 CLOSED http://bugs.python.org/issue8459 created haypo buildbot Set a timeout in test_urllib2net 2010-04-19 CLOSED http://bugs.python.org/issue8460 created haypo patch PythonLauncher universal build fails due to missing -arch and 2010-04-19 CLOSED http://bugs.python.org/issue8461 created ned.deily raise test_support.TestSkipped() is used outside main() / test 2010-04-19 http://bugs.python.org/issue8462 created haypo shutil.make_archive() supports bz2, but it's not documented 2010-04-19 CLOSED http://bugs.python.org/issue8463 created mastrodomenico patch tarfile creates tarballs with execute permissions set 2010-04-19 http://bugs.python.org/issue8464 created mastrodomenico patch re: Backreferences vs. escapes: a silent failure solved 2010-04-19 http://bugs.python.org/issue8465 created Aaron.Sherman typo in tp_name of cStringIO 2010-04-20 CLOSED http://bugs.python.org/issue8466 created anthon subprocess: surrogates of the error message (Python implementa 2010-04-20 http://bugs.python.org/issue8467 created haypo patch bz2: support surrogates in filename, and bytes/bytearray filen 2010-04-20 CLOSED http://bugs.python.org/issue8468 created haypo patch struct - please make sizes explicit 2010-04-20 http://bugs.python.org/issue8469 created kiilerix Let cmd.cmd.intro be unicode friendly 2010-04-20 http://bugs.python.org/issue8470 created phep Unicode returns in doctest can make subsequent tests fail 2010-04-20 http://bugs.python.org/issue8471 created lregebro patch itertools.filterfalse() function missing 2010-04-20 http://bugs.python.org/issue8472 reopened honeyman doctest fails if you have inconsistent lineendings 2010-04-20 http://bugs.python.org/issue8473 created lregebro patch Duplicate tests in email test suite 2010-04-20 CLOSED http://bugs.python.org/issue8474 created l0nwlf patch build-installer fix for doc building on OSX 10.4 2010-04-20 CLOSED http://bugs.python.org/issue8475 created db3l patch build-installer fix for setIcon when no script path specified 2010-04-20 http://bugs.python.org/issue8476 created db3l patch _ssl: support surrogates in filenames, and bytes/bytearray fil 2010-04-20 http://bugs.python.org/issue8477 created haypo patch tokenize.untokenize first token missing failure case 2010-04-21 http://bugs.python.org/issue8478 created rb test_gdb in Python3: No stack 2010-04-21 CLOSED http://bugs.python.org/issue8479 created loewis test_gdb: No frame is currently selected. 2010-04-21 CLOSED http://bugs.python.org/issue8480 created loewis doc: ctypes no need to explicitly allocate writable memory wit 2010-04-21 http://bugs.python.org/issue8481 created techtonik test_gdb, gdb/libpython.py: Unable to read information on pyth 2010-04-21 http://bugs.python.org/issue8482 created ncoghlan 64bit asyncore.dispatcher's __getattr__ method produces confusing ef 2010-04-21 http://bugs.python.org/issue8483 created djc patch ssl socket with certificate verification fails on SHA256 diges 2010-04-21 CLOSED http://bugs.python.org/issue8484 created beda patch Don't accept bytearray as filenames, or simplify the API 2010-04-21 CLOSED http://bugs.python.org/issue8485 created haypo patch threading.Event() 2010-04-21 CLOSED http://bugs.python.org/issue8486 created Kain94 os.mknod() fails on NFS mounted directories 2010-04-21 CLOSED http://bugs.python.org/issue8487 created Nikratio Docstrings of non-data descriptors "ignored" 2010-04-21 http://bugs.python.org/issue8488 created torsten smtplib.SMTP.login() breaks with unicode input 2010-04-21 http://bugs.python.org/issue8489 created iElectric asyncore test suite 2010-04-21 http://bugs.python.org/issue8490 created giampaolo.rodola patch, patch Need readline command and keybinding information 2010-04-21 http://bugs.python.org/issue8491 created MLModel Addition to readline module to get dictionary of keystrokes an 2010-04-21 http://bugs.python.org/issue8492 created MLModel socket's send can raise errno 35 under OS X, which causes prob 2010-04-21 http://bugs.python.org/issue8493 created mdcowles test_gdb assertEndsWith failing: Unable to read information on 2010-04-21 CLOSED http://bugs.python.org/issue8494 created loewis test_gdb: use utf8+surrogateescape charset? 2010-04-22 http://bugs.python.org/issue8495 created haypo patch mailcap.lookup() returns filter iterator rather than list if k 2010-04-22 CLOSED http://bugs.python.org/issue8496 created gnofi patch Technology and Licensing Related Query 2010-04-22 CLOSED http://bugs.python.org/issue8497 created renben Cannot use backlog = 0 for sockets 2010-04-22 http://bugs.python.org/issue8498 created Daniel.Evers Set a timeout in test_urllibnet 2010-04-22 http://bugs.python.org/issue8499 created haypo Erroneous Invalid Syntax Error 2010-04-22 CLOSED http://bugs.python.org/issue8500 created indolering --dry-run option doesn't work 2010-04-22 http://bugs.python.org/issue8501 created j1m proposal: encourage xgettext rather than pygettext.py in gette 2010-04-23 http://bugs.python.org/issue8502 created jhg patch smtpd SMTPServer does not allow domain filtering 2010-04-23 http://bugs.python.org/issue8503 created mike.s bsddb databases in python 2.6 are not compatible with python 2 2010-04-23 http://bugs.python.org/issue8504 created guy.linton 2to3 fix_future.py removes __future__ imports, but should it? 2010-04-23 http://bugs.python.org/issue8505 created barry SimpleXMLRPCServer Socket not closed after shutdown call 2010-04-23 http://bugs.python.org/issue8506 created othererik abc.abstractproperty does not copy docstring 2010-04-23 http://bugs.python.org/issue8507 created rescrv 2to3 fixer for gettext's .ugettext 2010-04-23 http://bugs.python.org/issue8508 created barry fix autoconf quoting in help strings and code snippets 2010-04-23 http://bugs.python.org/issue8509 created doko patch, needs review update to autoconf2.65 2010-04-23 http://bugs.python.org/issue8510 created doko patch, needs review Issues Now Closed (88) ______________________ asyncore and asynchat incompatible with Py3k str and bytes 869 days http://bugs.python.org/issue1563 r.david.murray __import__ with fromlist= 796 days http://bugs.python.org/issue2090 brett.cannon patch datetime: define division timedelta/timedelta 724 days http://bugs.python.org/issue2706 mark.dickinson patch RFC2732 support for urlparse (IPv6 addresses) 693 days http://bugs.python.org/issue2987 orsenthil patch, easy Allow application developers to select ciphers, and default to 610 days http://bugs.python.org/issue3597 pitrou patch Improve gdbinit of Python 2.6 607 days http://bugs.python.org/issue3631 haypo patch ftplib: ABOR does not consider 225 response code 586 days http://bugs.python.org/issue3817 giampaolo.rodola patch 26.rc1: test_signal issue on FreeBSD 6.3 584 days http://bugs.python.org/issue3864 r.david.murray easy, buildbot Support bytes for subprocess.Popen() 48 days http://bugs.python.org/issue4036 haypo patch, patch ftplib does not honour "timeout" parameter for active data con 472 days http://bugs.python.org/issue4814 giampaolo.rodola patch make distutils use shutil 470 days http://bugs.python.org/issue4843 merwok test_os causes delayed failure on x86 gentoo buildbot: Unknown 454 days http://bugs.python.org/issue4970 mark.dickinson patch, buildbot shutil.copystat does not copy the "hidden" flag on windows 437 days http://bugs.python.org/issue5168 tarek Document __instancecheck__ and __subclasscheck__ 430 days http://bugs.python.org/issue5250 cvrebert patch Obsolete RFCs should be removed from doc of urllib.urlparse 3 days http://bugs.python.org/issue5650 merwok patch, easy OS X Installer: add checks to ensure proper Tk configuration d 382 days http://bugs.python.org/issue5651 ronaldoussoren OS X Installer: remove references to Mac/Tools which no longer 382 days http://bugs.python.org/issue5652 ronaldoussoren shutil.copytree fails on dangling symlinks 272 days http://bugs.python.org/issue6547 TTimo MemoryError in AiX 64-bit - PyMem_MALLOC fails in open/fdopen 265 days http://bugs.python.org/issue6600 pitrou patch ftplib storelines does not honor strings returned in fp.readli 234 days http://bugs.python.org/issue6789 r.david.murray isspace(0xa0) is true on Mac OS X 194 days http://bugs.python.org/issue7072 ronaldoussoren patch, needs review urllib.request system proxy configuration lookup broken for OS 184 days http://bugs.python.org/issue7154 ronaldoussoren python script segment fault at PyMarshal_ReadLastObjectFromFil 157 days http://bugs.python.org/issue7332 Thomas.Smith patch Add {Create|Delete}KeyEx to _winreg, doc and test updates 154 days http://bugs.python.org/issue7347 brian.curtin patch, 64bit, needs review distutils makefile from python.org installer doesn't work on O 113 days http://bugs.python.org/issue7580 ronaldoussoren test_xmlrpc fails with non-ascii path 107 days http://bugs.python.org/issue7606 haypo patch httplib.HTTPConnection.getresponse closes socket which destroy 84 days http://bugs.python.org/issue7806 r.david.murray [decimal] ValueError -> TypeError in from_tuple 77 days http://bugs.python.org/issue7811 rhettinger SSL socket is not closed properly 65 days http://bugs.python.org/issue7927 pitrou test_platform failure on OS X 10.6 59 days http://bugs.python.org/issue7958 ronaldoussoren patch Backport capsule object 56 days http://bugs.python.org/issue7992 larry patch, buildbot os.stat on osx 10.5 or later doesn't expose all fields in stru 54 days http://bugs.python.org/issue8001 ronaldoussoren on OSX the file creation date not available in os.stat 54 days http://bugs.python.org/issue8002 ronaldoussoren utf8, backslashreplace and surrogates 45 days http://bugs.python.org/issue8092 haypo patch Support for TLS Server Name Indication extension 41 days http://bugs.python.org/issue8109 pitrou mywrite() ignores PyFile_WriteString() errors 42 days http://bugs.python.org/issue8124 haypo patch 2.6.5 OS X 10.5 --with-universal-archs=all (4-way) fails build 30 days http://bugs.python.org/issue8175 ronaldoussoren Crash in sqlite3.create_collation() with a string non encodabl 31 days http://bugs.python.org/issue8195 haypo patch regrtest stops prematurately on FreeBSD buildbot, with "succes 23 days http://bugs.python.org/issue8263 r.david.murray patch, buildbot python-gdb PyListTests fail 17 days http://bugs.python.org/issue8279 haypo patch test_gdb_sample fails 16 days http://bugs.python.org/issue8281 haypo patch, buildbot test_ssl failures with OpenSSL 1.0.0 12 days http://bugs.python.org/issue8322 pitrou patch diff.py produce unified format by default 9 days http://bugs.python.org/issue8355 techtonik patch OS X universal builds fail on 2.7b1 and py3k with "Don't know 8 days http://bugs.python.org/issue8366 ronaldoussoren Port of the gdb7 debugging hooks to the "py3k" branch 9 days http://bugs.python.org/issue8380 loewis patch subprocess: support undecodable current working directory on P 2 days http://bugs.python.org/issue8393 loewis patch ctypes.dlopen() doesn't support surrogates 1 days http://bugs.python.org/issue8394 loewis patch shutil.copytree() add a copy_func parameter. 6 days http://bugs.python.org/issue8395 tarek namedtuple vs tuple 3 days http://bugs.python.org/issue8415 rhettinger bytes and bytearray constructors raise TypeError for very larg 1 days http://bugs.python.org/issue8417 benjamin.peterson tiger buildbot: unable to resolv hostname address 0 days http://bugs.python.org/issue8421 r.david.murray tiger buildbot: test_abspath_issue3426 failure (test_genericpa 2 days http://bugs.python.org/issue8422 haypo buildbot test_site failure with non-ASCII directory 1 days http://bugs.python.org/issue8430 haypo patch, buildbot buildbot: test_gdb failure on sparc Ubuntu trunk 0 days http://bugs.python.org/issue8434 haypo buildbot It is possible to observe a mutating frozenset 1 days http://bugs.python.org/issue8435 rhettinger set.__init__ accepts keyword args 1 days http://bugs.python.org/issue8436 rhettinger easy test_gdb: gdb.Frame has no attribute function 3 days http://bugs.python.org/issue8437 ncoghlan patch Codecs: "surrogateescape" error handler in Python 2.7 1 days http://bugs.python.org/issue8438 pitrou patch test_linecache failing in py3k r80169 3 days http://bugs.python.org/issue8439 pitrou patch Framework build broken in 3.2 brunk 0 days http://bugs.python.org/issue8441 ronaldoussoren Broken zipfile with python 3.2 on osx 0 days http://bugs.python.org/issue8443 loewis syslog.syslog('msg') logs message with ident "python". 4 days http://bugs.python.org/issue8451 jafo patch, easy, needs review build_installer.py breaks bzip compilation 1 days http://bugs.python.org/issue8453 r.david.murray buildbot: test_select failure (test_returned_list_identity) on 1 days http://bugs.python.org/issue8459 benjamin.peterson buildbot Set a timeout in test_urllib2net 1 days http://bugs.python.org/issue8460 orsenthil patch PythonLauncher universal build fails due to missing -arch and 1 days http://bugs.python.org/issue8461 ronaldoussoren shutil.make_archive() supports bz2, but it's not documented 0 days http://bugs.python.org/issue8463 tarek patch typo in tp_name of cStringIO 0 days http://bugs.python.org/issue8466 eric.smith bz2: support surrogates in filename, and bytes/bytearray filen 3 days http://bugs.python.org/issue8468 haypo patch Duplicate tests in email test suite 2 days http://bugs.python.org/issue8474 ezio.melotti patch build-installer fix for doc building on OSX 10.4 2 days http://bugs.python.org/issue8475 loewis patch test_gdb in Python3: No stack 0 days http://bugs.python.org/issue8479 dmalcolm test_gdb: No frame is currently selected. 1 days http://bugs.python.org/issue8480 haypo ssl socket with certificate verification fails on SHA256 diges 1 days http://bugs.python.org/issue8484 pitrou patch Don't accept bytearray as filenames, or simplify the API 1 days http://bugs.python.org/issue8485 haypo patch threading.Event() 0 days http://bugs.python.org/issue8486 r.david.murray os.mknod() fails on NFS mounted directories 0 days http://bugs.python.org/issue8487 georg.brandl test_gdb assertEndsWith failing: Unable to read information on 1 days http://bugs.python.org/issue8494 dmalcolm mailcap.lookup() returns filter iterator rather than list if k 1 days http://bugs.python.org/issue8496 pitrou patch Technology and Licensing Related Query 0 days http://bugs.python.org/issue8497 brian.curtin Erroneous Invalid Syntax Error 0 days http://bugs.python.org/issue8500 pitrou Semaphore.acquire() timeout parameter 2332 days http://bugs.python.org/issue850728 pitrou patch Better SSL support in socket module 2145 days http://bugs.python.org/issue967275 pitrou ftplib PASV error bug 2026 days http://bugs.python.org/issue1037516 giampaolo.rodola shutil.move() does not preserve ownership 1619 days http://bugs.python.org/issue1355826 tarek easy Allow choice of copy function in shutil.copytree 1344 days http://bugs.python.org/issue1540112 tarek patch, easy datetime module missing some important methods 1144 days http://bugs.python.org/issue1673409 mark.dickinson patch ftplib and ProFTPD NLST 226 without 1xx response 1057 days http://bugs.python.org/issue1726451 giampaolo.rodola patch Top Issues Most Discussed (10) ______________________________ 26 python script segment fault at PyMarshal_ReadLastObjectFromFile 157 days closed http://bugs.python.org/issue7332 24 test_gdb: gdb.Frame has no attribute function 3 days closed http://bugs.python.org/issue8437 16 Test assumptions for test_itimer_virtual and test_itimer_prof 7 days open http://bugs.python.org/issue8424 15 optparse/argparse: provide a simple way to get a programmatical 536 days open http://bugs.python.org/issue4256 14 Codecs: "surrogateescape" error handler in Python 2.7 1 days closed http://bugs.python.org/issue8438 14 Improve GIL in 2.7 20 days open http://bugs.python.org/issue8299 14 curses crash on FreeBSD 151 days open http://bugs.python.org/issue7384 13 datetime: define division timedelta/timedelta 724 days closed http://bugs.python.org/issue2706 12 test_gdb, gdb/libpython.py: Unable to read information on pytho 2 days open http://bugs.python.org/issue8482 12 shutil.copytree fails on dangling symlinks 272 days closed http://bugs.python.org/issue6547 From barry at python.org Fri Apr 23 18:21:11 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 23 Apr 2010 12:21:11 -0400 Subject: [Python-Dev] autoconf update to 2.65 and cleanup In-Reply-To: <201004231744.21040.victor.stinner@haypocalc.com> References: <4BD1BC43.5020707@ubuntu.com> <201004231744.21040.victor.stinner@haypocalc.com> Message-ID: <20100423122111.38ff3dde@heresy> On Apr 23, 2010, at 05:44 PM, Victor Stinner wrote: >I'm not sure that it's a good idea to change the build process after the >first beta. It would depend on the issue comments ;-) OTOH, this doesn't seem like a new feature, so I think it should be okay. Doubly so if it fixes a bug. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Fri Apr 23 18:32:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Apr 2010 02:32:44 +1000 Subject: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals()) In-Reply-To: <85sk6mffq4.fsf@jobh.org> References: <85sk6mffq4.fsf@jobh.org> Message-ID: <4BD1CBAC.2050307@gmail.com> Joachim B Haga wrote: > There seem to be an inconsistency in the handling of local scopes in > exec. Consider the following code, which raises NameError if the '#' is > removed from the second last line. > > > block = """ > b = 'ok' > def f(): > print(b) # raises NameError here > f() > """ > scope = locals()#.copy() > exec(block, globals(), scope) > > > The intermediate scope is searched for the variable name if the third > argument to exec() is locals(), but not if it is locals().copy(). > Testing further, it looks like NameError is raised for any dict which > is not identically equal to either globals() or locals(). What actually matters is whether or not the first and second scope are the same dictionary or not. If they're different, then the supplied local scope is treated as equivalent to a class definition scope, and hence won't participate in lexical scoping. If they're the same (which happens implicitly if the second one is omitted) then they're treated as a module scope (and hence written values are visible as globals inside any defined functions). (Using 2.x syntax) >>> outer_scope = dict() >>> inner_scope = dict() >>> block = """ ... b = 'ok' ... def f(): ... print (b) ... f() ... """ >>> exec block in outer_scope ok >>> outer_scope.clear() >>> exec block in outer_scope, outer_scope ok >>> outer_scope.clear() >>> exec block in outer_scope, inner_scope Traceback (most recent call last): File "", line 1, in File "", line 5, in File "", line 4, in f NameError: global name 'b' is not defined Since changing this would break class definitions, that ain't going to happen. Suggestions for how to explain the behaviour more clearly in the exec() documentation probably wouldn't hurt though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From sridharr at activestate.com Fri Apr 23 18:42:50 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 23 Apr 2010 09:42:50 -0700 Subject: [Python-Dev] code.python.org - random 403 errors In-Reply-To: <20100423055521.GC3225@nexus.in-nomine.org> References: <8A563004-D8C7-4EB4-B5D1-A37AE1AD5FE9@activestate.com> <20100422210351.GA16581@yoda.bytereef.org> <4BD0C72D.6090609@v.loewis.de> <3578E3A2-2CBD-4737-BAB1-EFF4B0894F80@activestate.com> <20100423055521.GC3225@nexus.in-nomine.org> Message-ID: <855C211B-67EA-46D3-AFFE-2FA22A858DBB@activestate.com> On 2010-04-22, at 10:55 PM, Jeroen Ruigrok van der Werven wrote: > -On [20100423 02:48], Sridhar Ratnakumar (sridharr at activestate.com) wrote: >>> Ok, I setup a cron job to maintain an internal mirror of the above mentioned repositories in code.python.org. We'll do a "hg pull -u" (equivalent to "svn up") every hour; no clones. Hopefully, that should reduce the amount of requests from our side. Let me know if in future this issue repeats. > > Dirk Jan can probably correct me (or some other heavy Hg user) but for all I > know you should indeed simply clone once and subsequently hg pull and from > your local copy clone as you like. (At least that's also how > http://wiki.services.openoffice.org/wiki/Mercurial/Getting_Started seems to > aim at explaining.) Since the "download Python source code" thing is just part of the ActivePython build script, and the Hudson build script deletes the "build/" directory of the previous build .. a clone was necessary. To fix this I ended up creating a "mirror" at a local site which mirror was maintained by an hourly 'hg pull -u'. The hudson build script still does a clone although from this local mirror URL. (Incidentally, cloning from this mirror via the Apache index listing URL doesn't seem to work; gotta investigate why...) -srid From dgindikin at gmail.com Fri Apr 23 20:11:53 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 18:11:53 +0000 (UTC) Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution Message-ID: We were having performance problems unpickling a large pickle file, we were getting 170s running time (which was fine), but 1100mb memory usage. Memory usage ought to have been about 300mb, this was happening because of memory fragmentation, due to many unnecessary "puts" in the pickle stream. We made a pickletools.optimize inspired tool that could run directly on a pickle file and used pickletools.genops. This solved the unpickling problem (84s, 382mb). However the tool itself was using too much memory and time (1100s, 470mb), so I recoded it to scan through the pickle stream directly, without going through pickletools.genops, giving (240s, 130mb). Other people that deal with large pickle files are probably having similar problems, and since this comes up when dealing with large data it is precisely in this situation that you probably can't use pickletools.optimize or pickletools.genops. It feels like functionality that ought to be added to pickletools, is there some way I can contribute this? From brett at python.org Fri Apr 23 20:24:05 2010 From: brett at python.org (Brett Cannon) Date: Fri, 23 Apr 2010 11:24:05 -0700 Subject: [Python-Dev] autoconf update to 2.65 and cleanup In-Reply-To: <20100423122111.38ff3dde@heresy> References: <4BD1BC43.5020707@ubuntu.com> <201004231744.21040.victor.stinner@haypocalc.com> <20100423122111.38ff3dde@heresy> Message-ID: On Fri, Apr 23, 2010 at 09:21, Barry Warsaw wrote: > On Apr 23, 2010, at 05:44 PM, Victor Stinner wrote: > > >I'm not sure that it's a good idea to change the build process after the > >first beta. It would depend on the issue comments ;-) > > OTOH, this doesn't seem like a new feature, so I think it should be okay. > Doubly so if it fixes a bug. > > I'm with Barry; if this fixes something then it's worth a go. -Brett > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Apr 23 20:27:09 2010 From: brett at python.org (Brett Cannon) Date: Fri, 23 Apr 2010 11:27:09 -0700 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 11:11, Dan Gindikin wrote: > We were having performance problems unpickling a large pickle file, we were > getting 170s running time (which was fine), but 1100mb memory usage. Memory > usage ought to have been about 300mb, this was happening because of memory > fragmentation, due to many unnecessary "puts" in the pickle stream. > > We made a pickletools.optimize inspired tool that could run directly on a > pickle file and used pickletools.genops. This solved the unpickling problem > (84s, 382mb). > > However the tool itself was using too much memory and time (1100s, 470mb), > so > I recoded it to scan through the pickle stream directly, without going > through > pickletools.genops, giving (240s, 130mb). > > Other people that deal with large pickle files are probably having similar > problems, and since this comes up when dealing with large data it is > precisely > in this situation that you probably can't use pickletools.optimize or > pickletools.genops. It feels like functionality that ought to be added to > pickletools, is there some way I can contribute this? > The best next step is to open an issue at bugs.python.org and upload the patch. I can't make any guarantees on when someone will look at it or if it will get accepted, but putting the code there is your best bet for acceptance. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre at peadrop.com Fri Apr 23 20:38:16 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Fri, 23 Apr 2010 14:38:16 -0400 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 2:11 PM, Dan Gindikin wrote: > We were having performance problems unpickling a large pickle file, we were > getting 170s running time (which was fine), but 1100mb memory usage. Memory > usage ought to have been about 300mb, this was happening because of memory > fragmentation, due to many unnecessary "puts" in the pickle stream. > > We made a pickletools.optimize inspired tool that could run directly on a > pickle file and used pickletools.genops. This solved the unpickling problem > (84s, 382mb). > > However the tool itself was using too much memory and time (1100s, 470mb), so > I recoded it to scan through the pickle stream directly, without going through > pickletools.genops, giving (240s, 130mb). > Collin Winter wrote a simple optimization pass for cPickle in Unladen Swallow [1]. The code reads through the stream and remove all the unnecessary PUTs in-place. [1]: http://code.google.com/p/unladen-swallow/source/browse/trunk/Modules/cPickle.c#735 > Other people that deal with large pickle files are probably having similar > problems, and since this comes up when dealing with large data it is precisely > in this situation that you probably can't use pickletools.optimize or > pickletools.genops. It feels like functionality that ought to be added to > pickletools, is there some way I can contribute this? > Just put your code on bugs.python.org and I will take a look. -- Alexandre From alexandre at peadrop.com Fri Apr 23 20:49:47 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Fri, 23 Apr 2010 14:49:47 -0400 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti wrote: > Collin Winter wrote a simple optimization pass for cPickle in Unladen > Swallow [1]. The code reads through the stream and remove all the > unnecessary PUTs in-place. > I just noticed the code removes *all* PUT opcodes, regardless if they are needed or not. So, this code can only be used if there's no GET in the stream (which is unlikely for a large stream). I believe Collin made this trade-off for performance reasons. However, it wouldn't be hard to make the current code to work like pickletools.optimize(). -- Alexandre From dgindikin at gmail.com Fri Apr 23 20:53:56 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 18:53:56 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Alexandre Vassalotti peadrop.com> writes: > Just put your code on bugs.python.org and I will take a look. > Thanks, I'll put it in there. From collinwinter at google.com Fri Apr 23 20:53:57 2010 From: collinwinter at google.com (Collin Winter) Date: Fri, 23 Apr 2010 11:53:57 -0700 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti wrote: > On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti > wrote: >> Collin Winter wrote a simple optimization pass for cPickle in Unladen >> Swallow [1]. The code reads through the stream and remove all the >> unnecessary PUTs in-place. >> > > I just noticed the code removes *all* PUT opcodes, regardless if they > are needed or not. So, this code can only be used if there's no GET in > the stream (which is unlikely for a large stream). I believe Collin > made this trade-off for performance reasons. However, it wouldn't be > hard to make the current code to work like pickletools.optimize(). The optimization pass is only run if you don't use any GETs. The optimization is also disabled if you're writing to a file-like object. These tradeoffs were appropriate for the workload I was optimizing against. Collin Winter From collinwinter at google.com Fri Apr 23 21:07:28 2010 From: collinwinter at google.com (Collin Winter) Date: Fri, 23 Apr 2010 12:07:28 -0700 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 11:53 AM, Collin Winter wrote: > On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti > wrote: >> On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti >> wrote: >>> Collin Winter wrote a simple optimization pass for cPickle in Unladen >>> Swallow [1]. The code reads through the stream and remove all the >>> unnecessary PUTs in-place. >>> >> >> I just noticed the code removes *all* PUT opcodes, regardless if they >> are needed or not. So, this code can only be used if there's no GET in >> the stream (which is unlikely for a large stream). I believe Collin >> made this trade-off for performance reasons. However, it wouldn't be >> hard to make the current code to work like pickletools.optimize(). > > The optimization pass is only run if you don't use any GETs. The > optimization is also disabled if you're writing to a file-like object. > These tradeoffs were appropriate for the workload I was optimizing > against. I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling benchmarks in the benchmark suite should help assess this. The current optimization penalizes pickling to speed up unpickling, which made sense when optimizing pickles that would go into memcache and be read out 13-15x more often than they were written. Collin Winter From alexandre at peadrop.com Fri Apr 23 21:24:54 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Fri, 23 Apr 2010 15:24:54 -0400 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 3:07 PM, Collin Winter wrote: > I should add that, adding the necessary bookkeeping to remove only > unused PUTs (instead of the current all-or-nothing scheme) should not > be hard. I'd watch out for a further performance/memory hit; the > pickling benchmarks in the benchmark suite should help assess this. I was thinking about this too. A simple boolean table could be fast, while keeping the space requirement down. This scheme would be nice to caches as well. > The current optimization penalizes pickling to speed up unpickling, > which made sense when optimizing pickles that would go into memcache > and be read out 13-15x more often than they were written. This is my current impression of how pickle is most often used. Are you aware of a use case of pickle where you do more writes than reads? I can't think of any. -- Alexandre From jobh at broadpark.no Fri Apr 23 21:53:51 2010 From: jobh at broadpark.no (Joachim B Haga) Date: Fri, 23 Apr 2010 21:53:51 +0200 Subject: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals()) References: <85sk6mffq4.fsf@jobh.org> <4BD1CBAC.2050307@gmail.com> Message-ID: <85iq7ihrn4.fsf@jobh.org> Nick Coghlan writes: > Joachim B Haga wrote: >> There seem to be an inconsistency in the handling of local scopes in >> exec. [...] >> >> The intermediate scope is searched for the variable name if the third >> argument to exec() is locals(), but not if it is locals().copy(). > > What actually matters is whether or not the first and second scope are > the same dictionary or not. > > If they're different, then the supplied local scope is treated as > equivalent to a class definition scope, and hence won't participate in > lexical scoping. If they're the same (which happens implicitly if the > second one is omitted) then they're treated as a module scope (and hence > written values are visible as globals inside any defined functions). Ok, thank you for the explanation. > Since changing this would break class definitions, that ain't going to > happen. Suggestions for how to explain the behaviour more clearly in the > exec() documentation probably wouldn't hurt though. I don't quite see how exec() affects the class definition syntax? Anyhow, I definitely agree that this should be documented. I suggest the following (condensed from your explanation): -If provided, /locals/ can be any mapping object. +If provided, /locals/ can be any mapping object. It is treated as equivalent to a class definition scope, and hence does not participate in lexical scoping. -- Joachim B Haga From dgindikin at gmail.com Fri Apr 23 21:57:45 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 19:57:45 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Collin Winter google.com> writes: > I should add that, adding the necessary bookkeeping to remove only > unused PUTs (instead of the current all-or-nothing scheme) should not > be hard. I'd watch out for a further performance/memory hit; the > pickling benchmarks in the benchmark suite should help assess this. > The current optimization penalizes pickling to speed up unpickling, > which made sense when optimizing pickles that would go into memcache > and be read out 13-15x more often than they were written. This wouldn't help our use case, your code needs the entire pickle stream to be in memory, which in our case would be about 475mb, this is on top of the 300mb+ data structures that generated the pickle stream. From alexandre at peadrop.com Fri Apr 23 22:53:52 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Fri, 23 Apr 2010 16:53:52 -0400 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin wrote: > This wouldn't help our use case, your code needs the entire pickle > stream to be in memory, which in our case would be about 475mb, this > is on top of the 300mb+ data structures that generated the pickle > stream. > In that case, the best we could do is a two-pass algorithm to remove the unused PUTs. That won't be efficient, but it will satisfy the memory constraint. Another solution is to not generate the PUTs at all by setting the 'fast' attribute on Pickler. But that won't work if you have a recursive structure, or have code that requires that the identity of objects to be preserved. >>> import io, pickle >>> x=[1,2] >>> f = io.BytesIO() >>> p = pickle.Pickler(f, protocol=-1) >>> p.dump([x,x]) >>> pickletools.dis(f.getvalue()) 0: \x80 PROTO 2 2: ] EMPTY_LIST 3: q BINPUT 0 5: ( MARK 6: ] EMPTY_LIST 7: q BINPUT 1 9: ( MARK 10: K BININT1 1 12: K BININT1 2 14: e APPENDS (MARK at 9) 15: h BINGET 1 17: e APPENDS (MARK at 5) 18: . STOP highest protocol among opcodes = 2 >>> [id(x) for x in pickle.loads(f.getvalue())] [20966504, 20966504] Now with the 'fast' mode enabled: >>> f = io.BytesIO() >>> p = pickle.Pickler(f, protocol=-1) >>> p.fast = True >>> p.dump([x,x]) >>> pickletools.dis(f.getvalue()) 0: \x80 PROTO 2 2: ] EMPTY_LIST 3: ( MARK 4: ] EMPTY_LIST 5: ( MARK 6: K BININT1 1 8: K BININT1 2 10: e APPENDS (MARK at 5) 11: ] EMPTY_LIST 12: ( MARK 13: K BININT1 1 15: K BININT1 2 17: e APPENDS (MARK at 12) 18: e APPENDS (MARK at 3) 19: . STOP highest protocol among opcodes = 2 >>> [id(x) for x in pickle.loads(f.getvalue())] [20966504, 21917992] As you can observe, the pickle stream generated with the fast mode might actually be bigger. By the way, it is weird that the total memory usage of the data structure is smaller than the size of its respective pickle stream. What pickle protocol are you using? -- Alexandre From dgindikin at gmail.com Fri Apr 23 23:11:34 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 21:11:34 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Alexandre Vassalotti peadrop.com> writes: > > On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin gmail.com> wrote: > > This wouldn't help our use case, your code needs the entire pickle > > stream to be in memory, which in our case would be about 475mb, this > > is on top of the 300mb+ data structures that generated the pickle > > stream. > > > > In that case, the best we could do is a two-pass algorithm to remove > the unused PUTs. That won't be efficient, but it will satisfy the > memory constraint. That is for what I'm doing for us right now. > Another solution is to not generate the PUTs at all > by setting the 'fast' attribute on Pickler. But that won't work if you > have a recursive structure, or have code that requires that the > identity of objects to be preserved. We definitely have some cross links amongst the objects, so we need PUTs. > By the way, it is weird that the total memory usage of the data > structure is smaller than the size of its respective pickle stream. > What pickle protocol are you using? Its highest protocol, but we have a bunch of extension types that get expanded into python tuples for pickling. From collinwinter at google.com Fri Apr 23 23:18:13 2010 From: collinwinter at google.com (Collin Winter) Date: Fri, 23 Apr 2010 14:18:13 -0700 Subject: [Python-Dev] Unpickling memory usage problem, and a proposed solution In-Reply-To: References: Message-ID: On Fri, Apr 23, 2010 at 1:53 PM, Alexandre Vassalotti wrote: > On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin wrote: >> This wouldn't help our use case, your code needs the entire pickle >> stream to be in memory, which in our case would be about 475mb, this >> is on top of the 300mb+ data structures that generated the pickle >> stream. >> > > In that case, the best we could do is a two-pass algorithm to remove > the unused PUTs. That won't be efficient, but it will satisfy the > memory constraint. Another solution is to not generate the PUTs at all > by setting the 'fast' attribute on Pickler. But that won't work if you > have a recursive structure, or have code that requires that the > identity of objects to be preserved. I don't think it's possible in general to remove any PUTs if the pickle is being written to a file-like object. It is possible to reuse a single Pickler to pickle multiple objects: this causes the Pickler's memo dict to be shared between the objects being pickled. If you pickle foo, bar, and baz, foo may not have any GETs, but bar and baz may have GETs that reference data added to the memo by foo's PUT operations. Because you can't know what will be written to the file-like object later, you can't remove any of the PUT instructions in this scenario. This kind of thing is done in real-world code like cvs2svn (which I broke when I was optimizing cPickle; don't break cvs2svn, it's not fun to fix :). I added some basic tests for this support in cPython's Lib/test/pickletester.py. There might be room for app-specific optimizations that do this, but I'm not sure it would work for a general-usage cPickle that needs to stay compatible with the current system. Collin Winter From solipsis at pitrou.net Fri Apr 23 23:24:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Apr 2010 21:24:10 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Collin Winter google.com> writes: > > I don't think it's possible in general to remove any PUTs if the > pickle is being written to a file-like object. Does cPickle bytecode have some kind of NOP instruction? You could keep track of which PUTs weren't necessary and zero them out at the end. It would be much cheaper than writing a whole other "optimized" stream. (of course it only works on a seekable stream :-)) Regards Antoine. From dgindikin at gmail.com Fri Apr 23 23:44:50 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 21:44:50 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Collin Winter google.com> writes: > I don't think it's possible in general to remove any PUTs if the > pickle is being written to a file-like object. It is possible to reuse > a single Pickler to pickle multiple objects: this causes the Pickler's > memo dict to be shared between the objects being pickled. If you > pickle foo, bar, and baz, foo may not have any GETs, but bar and baz > may have GETs that reference data added to the memo by foo's PUT > operations. Because you can't know what will be written to the > file-like object later, you can't remove any of the PUT instructions > in this scenario. Hmm, that is a good point. A possible solution would be for the two-pass optimizer to scan through the entire file, going right through '.' opcodes. That would deal with the case you are describing, but not if the user "maliciously" wrote some other stuff into the file in between pickle dumps, all the while reusing the same pickler. I think a better solution would be to make sure that the '.' is the last thing in the file and die otherwise. This would at least ensure correctness and detection of cases that this thing could not handle. > don't break cvs2svn, it's not fun > to fix :). I added some basic tests for this support in cPython's > Lib/test/pickletester.py. Thanks for the warning :) > There might be room for app-specific optimizations that do this, but > I'm not sure it would work for a general-usage cPickle that needs to > stay compatible with the current system. That may well be true. Still, when trying to deal with large data you really need something like this. Our situation was made worse because we had a extension types. As they were allocated they got interspersed with temporaries generated by the spurious PUTs, and that is what really fragmented the memory. However its probably not a stretch to assume that if you are dealing with large stuff through python you are going to have extension types in the mix. From dgindikin at gmail.com Fri Apr 23 23:50:50 2010 From: dgindikin at gmail.com (Dan Gindikin) Date: Fri, 23 Apr 2010 21:50:50 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Antoine Pitrou pitrou.net> writes: > Does cPickle bytecode have some kind of NOP instruction? > You could keep track of which PUTs weren't necessary and zero them out at the > end. It would be much cheaper than writing a whole other "optimized" stream. For a large file, I'm not sure it is much faster to edit it in place than to rewrite it, and also since it's a large file, you are going to probably have it compressed, in which case you are out of luck anyway. > (of course it only works on a seekable stream ) Indeed... since I was dealing with zipped files, I had to pass in a "seek0" func, like so: file_in_seek0_func = lambda x: os.popen('zcat ' + file_in) From solipsis at pitrou.net Fri Apr 23 23:56:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Apr 2010 21:56:19 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Unpickling_memory_usage_problem=2C=09and_a?= =?utf-8?q?_proposed_=09solution?= References: Message-ID: Dan Gindikin gmail.com> writes: > > Antoine Pitrou pitrou.net> writes: > > Does cPickle bytecode have some kind of NOP instruction? > > You could keep track of which PUTs weren't necessary and zero them out at the > > end. It would be much cheaper than writing a whole other "optimized" stream. > > For a large file, I'm not sure it is much faster to edit it in place > than to rewrite it, and also since it's a large file, you are going > to probably have it compressed, in which case you are out of luck anyway. Depends whether you really care about disk occupation or not. Disk space is often cheap (much more so than RAM). Also, I'm quite sure overwriting a couple of blocks in a file is cheaper than rewriting it entirely. Of course, if you must overwrite every other block, it might not be true anymore :-) Regards Antoine. From ncoghlan at gmail.com Sat Apr 24 10:38:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Apr 2010 18:38:55 +1000 Subject: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals()) In-Reply-To: <85iq7ihrn4.fsf@jobh.org> References: <85sk6mffq4.fsf@jobh.org> <4BD1CBAC.2050307@gmail.com> <85iq7ihrn4.fsf@jobh.org> Message-ID: <4BD2AE1F.6070906@gmail.com> Joachim B Haga wrote: >> Since changing this would break class definitions, that ain't going to >> happen. Suggestions for how to explain the behaviour more clearly in the >> exec() documentation probably wouldn't hurt though. > > I don't quite see how exec() affects the class definition syntax? I was merely pointing out that running exec() with separate globals and locals namespaces ends up invoking the same underlying machinery as is used to execute a class definition. For the docs suggestions, could you either email that to docs at python.org, or else create a documentation issue at bugs.python.org? (It will get lost otherwise) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Sat Apr 24 18:15:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 24 Apr 2010 17:15:35 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> Message-ID: <4BD31927.4010308@voidspace.org.uk> On 18/04/2010 15:13, Ronald Oussoren wrote: > On 14 Apr, 2010, at 23:37, Michael Foord wrote: > > >> On 14/04/2010 23:32, Greg Ewing wrote: >> >>> Michael Foord wrote: >>> >>>> Building Python requires, I believe, the XCode development tools to be installed. Even then, building a full version of Python - with *all* the C extensions that are part of a Python release - is not a trivial task. >>>> >>> What's non-trivial about it? I usually find that the normal >>> "./configure; make; make install" sequence works fine without >>> any further intervention. >>> >>> If you want a framework installation you have to read the >>> README and use a couple of extra options, but it still works >>> very smoothly. >>> >>> >> A build on my machine produces output similar to: >> >> >> Python build finished, but the necessary bits to build these modules were not found: >> _bsddb dl gdbm >> imageop linuxaudiodev ossaudiodev >> readline spwd sunaudiodev >> To find the necessary bits, look in setup.py in detect_modules() for the module's name. >> >> >> Failed to build these modules: >> _tkinter >> >> Obviously many of those are not meant to be built and I usually build Python for running the test suite - so I don't care about not having Tkinter. A new user of Python would most certainly care about not having Tkinter. >> > > What's the OS version? Do you have a copy of Tcl/Tk in /Library/Frameworks? > 10.6.3 and yes I have Tcl and Tk in /Library/Frameworks. How do I determine which versions they are? All the best, Michael > Ronald > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From db3l.net at gmail.com Sat Apr 24 22:34:18 2010 From: db3l.net at gmail.com (David Bolen) Date: Sat, 24 Apr 2010 16:34:18 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> Message-ID: Michael Foord writes: > 10.6.3 and yes I have Tcl and Tk in /Library/Frameworks. How do I > determine which versions they are? You can use "info patchlevel" in tclsh - assuming you're running a tclsh linked to your /Library version (a normal Tcl install puts this in /usr/local/bin I think). Or, tcl.h (in the Headers folder beneath the framework install) has TCL_VERSION and TCL_PATCH_LEVEL defines near the top of the file. Given that your error is a failure to build and not a skip, it sounds like setup is finding Tcl/Tk. From a quick glance, it looks like tkinter may also require the X11 headers (you'd have to have installed X11 separately) - do you have output in your log from what exactly is failing when that module attempts to build? -- David From fuzzyman at voidspace.org.uk Sat Apr 24 22:43:22 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 24 Apr 2010 21:43:22 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> Message-ID: <4BD357EA.5020604@voidspace.org.uk> On 24/04/2010 21:34, David Bolen wrote: > Michael Foord writes: > > >> 10.6.3 and yes I have Tcl and Tk in /Library/Frameworks. How do I >> determine which versions they are? >> > You can use "info patchlevel" in tclsh - assuming you're running a > tclsh linked to your /Library version (a normal Tcl install puts this > in /usr/local/bin I think). > > $ tclsh % info patchlevel 8.5.7 > Or, tcl.h (in the Headers folder beneath the framework install) has > TCL_VERSION and TCL_PATCH_LEVEL defines near the top of the file. > > Given that your error is a failure to build and not a skip, it sounds > like setup is finding Tcl/Tk. From a quick glance, it looks like > tkinter may also require the X11 headers (you'd have to have installed > X11 separately) - do you have output in your log from what exactly > is failing when that module attempts to build? > Hmmm... looks like a 32 / 64 bit issue, which I believe may be the expected result when trying to build on Snow Leopard (?). i686-apple-darwin10-gcc-4.2.1: -framework: linker input file unused because linking not done i686-apple-darwin10-gcc-4.2.1: Tk: linker input file unused because linking not done ld: warning: in /Library/Frameworks//Tcl.framework/Tcl, missing required architecture x86_64 in file ld: warning: in /Library/Frameworks//Tk.framework/Tk, missing required architecture x86_64 in file *** WARNING: renaming "_tkinter" since importing it failed: dlopen(build/lib.macosx-10.4-x86_64-2.7-pydebug/_tkinter.so, 2): Symbol not found: _TclFreeObj Referenced from: /compile/python-trunk/build/lib.macosx-10.4-x86_64-2.7-pydebug/_tkinter.so Expected in: dynamic lookup I think my Tk/Tcl install came from an Activestate installer. Michael > -- David > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From db3l.net at gmail.com Sat Apr 24 22:50:46 2010 From: db3l.net at gmail.com (David Bolen) Date: Sat, 24 Apr 2010 16:50:46 -0400 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> <4BD357EA.5020604@voidspace.org.uk> Message-ID: Michael Foord writes: > Hmmm... looks like a 32 / 64 bit issue, which I believe may be the > expected result when trying to build on Snow Leopard (?). I think so - I haven't tried a 64-bit build myself, but there's a comment in setup.py indicating that none of the Tcl/Tk framework builds support 64-bit. So I suppose you'd have to build 32-bit if you wanted a working _tkinter. -- David From fuzzyman at voidspace.org.uk Sat Apr 24 23:16:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 24 Apr 2010 22:16:46 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> <4BD357EA.5020604@voidspace.org.uk> Message-ID: <4BD35FBE.2030403@voidspace.org.uk> On 24/04/2010 21:50, David Bolen wrote: > Michael Foord writes: > > >> Hmmm... looks like a 32 / 64 bit issue, which I believe may be the >> expected result when trying to build on Snow Leopard (?). >> > I think so - I haven't tried a 64-bit build myself, but there's a > comment in setup.py indicating that none of the Tcl/Tk framework > builds support 64-bit. > > So I suppose you'd have to build 32-bit if you wanted a working > _tkinter. > > Sorry for my ignorance - how do I force a 32 bit build? Michael > -- David > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Sat Apr 24 23:56:00 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 24 Apr 2010 22:56:00 +0100 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BD35FBE.2030403@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> <4BD357EA.5020604@voidspace.org.uk> <4BD35FBE.2030403@voidspace.org.uk> Message-ID: <4BD368F0.4080100@voidspace.org.uk> On 24/04/2010 22:16, Michael Foord wrote: > On 24/04/2010 21:50, David Bolen wrote: >> Michael Foord writes: >>> Hmmm... looks like a 32 / 64 bit issue, which I believe may be the >>> expected result when trying to build on Snow Leopard (?). >> I think so - I haven't tried a 64-bit build myself, but there's a >> comment in setup.py indicating that none of the Tcl/Tk framework >> builds support 64-bit. >> >> So I suppose you'd have to build 32-bit if you wanted a working >> _tkinter. >> > Sorry for my ignorance - how do I force a 32 bit build? > Ok, so the following configure command line works and successfully builds Tkinter: ./configure --prefix=/dev/null --with-pydebug --enable-universalsdk All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ronaldoussoren at mac.com Sun Apr 25 16:33:30 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 25 Apr 2010 16:33:30 +0200 Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X) In-Reply-To: <4BD31927.4010308@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54E6B.3090209@v.loewis.de> <4BC54FD3.4030907@holdenweb.com> <4BC586BA.9010704@voidspace.org.uk> <4BC63457.5020006@canterbury.ac.nz> <4BC63599.5020005@voidspace.org.uk> <4BD31927.4010308@voidspace.org.uk> Message-ID: <244D9A86-7AB5-4C09-A851-379191E7E84E@mac.com> On 24 Apr, 2010, at 18:15, Michael Foord wrote: > On 18/04/2010 15:13, Ronald Oussoren wrote: >> On 14 Apr, 2010, at 23:37, Michael Foord wrote: >> >> >>> On 14/04/2010 23:32, Greg Ewing wrote: >>> >>>> Michael Foord wrote: >>>> >>>>> Building Python requires, I believe, the XCode development tools to be installed. Even then, building a full version of Python - with *all* the C extensions that are part of a Python release - is not a trivial task. >>>>> >>>> What's non-trivial about it? I usually find that the normal >>>> "./configure; make; make install" sequence works fine without >>>> any further intervention. >>>> >>>> If you want a framework installation you have to read the >>>> README and use a couple of extra options, but it still works >>>> very smoothly. >>>> >>>> >>> A build on my machine produces output similar to: >>> >>> >>> Python build finished, but the necessary bits to build these modules were not found: >>> _bsddb dl gdbm >>> imageop linuxaudiodev ossaudiodev >>> readline spwd sunaudiodev >>> To find the necessary bits, look in setup.py in detect_modules() for the module's name. >>> >>> >>> Failed to build these modules: >>> _tkinter >>> >>> Obviously many of those are not meant to be built and I usually build Python for running the test suite - so I don't care about not having Tkinter. A new user of Python would most certainly care about not having Tkinter. >>> >> >> What's the OS version? Do you have a copy of Tcl/Tk in /Library/Frameworks? >> > > 10.6.3 and yes I have Tcl and Tk in /Library/Frameworks. How do I determine which versions they are? $ ls -l /Library/Frameworks/Tcl.framework/Versions total 8 drwxr-xr-x 9 sysadmin admin 306 Oct 6 2009 8.5 lrwxr-xr-x 1 sysadmin admin 3 Oct 25 2009 Current -> 8.5 As you can see, my system has Tcl 8.5. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From torsten at debian.org Sun Apr 25 17:02:55 2010 From: torsten at debian.org (Torsten Landschoff) Date: Sun, 25 Apr 2010 17:02:55 +0200 Subject: [Python-Dev] Property inheritance in Python Message-ID: <20100425150255.GB27438@merzeus.obrandt.org> Hi Python experts. [It should be obvious, but you can run the code in this message via python -m doctest body.txt if you saved it as body.txt] In an application I develop on I want to use properties instead of the getter/setter paradigm. I ran into a problem overriding a property in a subclass. While I know how to do it with current Python, I propose changing the behaviour to be more consistent. For presentation, here is a stupid example using getters and setters: >>> class BaseGetterSetter(object): ... def get_p(self): ... return self._p ... def set_p(self, value): ... self._p = value >>> class DerivedGetterSetter(BaseGetterSetter): ... def get_p(self): ... return super(DerivedGetterSetter, self).get_p() * 2 ... def set_p(self, value): ... super(DerivedGetterSetter, self).set_p(value / 2) >>> d = DerivedGetterSetter() >>> d.set_p(42) >>> d._p 21 >>> d.get_p() 42 When translating this to use properties, I would come up with something like this: >>> class BaseProp(object): ... @property ... def p(self): ... return self._p ... @p.setter ... def p(self, value): ... self._p = value >>> class DerivedProp(BaseProp): ... @property ... def p(self): ... return super(DerivedProp, self).p * 2 ... @p.setter ... def p(self, value): ... super(DerivedProp, self).p = value / 2 >>> d = DerivedProp() >>> d._p = 21 >>> d.p 42 >>> d.p = 50 Traceback (most recent call last): ... AttributeError: 'super' object has no attribute 'p' As can be seen, using super like in the getter/setter approach above works for fget but not for fset. Support for using the getter via super() was added for Python 2.3 according to http://mail.python.org/pipermail/python-dev/2003-April/034702.html I think it would be more consistent to be able to access the __set__ call via super() as well. Working around -------------- The problematic code boils down to >>> super(DerivedProp, d).p = 1 Traceback (most recent call last): ... AttributeError: 'super' object has no attribute 'p' which can be worked around like this: >>> BaseProp.p.fset(d, 25) >>> BaseProp.p.__set__(d, 10) I'd rather use the method resolution order computed by Python so that mixin classes can extend the behaviour of properties. For that, one can extend super: >>> class duper(super): ... def __setattr__(self, name, value): ... mro = self.__self_class__.__mro__ ... for pos in xrange(len(mro)): ... if mro[pos] == self.__thisclass__: ... break ... for pos in xrange(pos+1, len(mro)): ... tmp = mro[pos] ... if isinstance(tmp, type) and name in tmp.__dict__: ... desc = tmp.__dict__[name] ... desc.__set__(self.__self__, value) ... return >>> duper(DerivedProp, d).p = 100 >>> d.p 200 Extending super --------------- I wrote a test case for the super() behaviour as I would like it to be and implemented the __setattr__ of duper above into super's C implementation. The code is not of production quality and there are some open questions. But I figured that I'd rather ask for opinions before spending even more time on this. The code is available on launchpad at this URL: https://code.launchpad.net/~torsten/python/descr-set-via-super What do you think, do you agree with my understanding or am I completely wrong? Hoping for some insightful comments, Torsten From jafo at tummy.com Sun Apr 25 22:31:22 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Sun, 25 Apr 2010 14:31:22 -0600 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. Message-ID: <4BD4A69A.4080507@tummy.com> I'm trying to get a good friend of mine to start doing bug triage on Python. As part of my trying to mentor him on it, I've found that many of the common things I do in triage, like setting a priority for priorityless bugs, assigning them to people who obviously are the next step, requires enhanced privileges. He has no reputation in the Python community, so I'd be up for getting him started on things that require fewer privileges like verifying older patches still apply against newer Pythons, or maybe summarizing priority/assignment changes to the list and having someone (possibly me) make the changes, etc... However, I will step up for him and say that I've known him a decade, and he's very trustworthy. He has been the president (we call that position Maximum Leader) of our Linux Users Group here for 5 years or so. Thoughts? Thanks, Sean -- Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: From skip at pobox.com Sun Apr 25 23:24:40 2010 From: skip at pobox.com (skip at pobox.com) Date: Sun, 25 Apr 2010 16:24:40 -0500 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: <4BD4A69A.4080507@tummy.com> References: <4BD4A69A.4080507@tummy.com> Message-ID: <19412.45848.576615.634858@montanaro.dyndns.org> Sean> However, I will step up for him and say that I've known him a Sean> decade, and he's very trustworthy. He has been the president (we Sean> call that position Maximum Leader) of our Linux Users Group here Sean> for 5 years or so. Given that Sean is vouching for him I'm fine with it. Skip From solipsis at pitrou.net Sun Apr 25 23:39:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Apr 2010 21:39:42 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > > Sean> However, I will step up for him and say that I've known him a > Sean> decade, and he's very trustworthy. He has been the president (we > Sean> call that position Maximum Leader) of our Linux Users Group here > Sean> for 5 years or so. > > Given that Sean is vouching for him I'm fine with it. I'm not sure I agree. Of course it could be argued the risk is minimal, but I think it's better if all people go through the same path of proving their motivation and quality of work. And if there's something wrong with that process we'd better address it than give random privileges to people we like :) Regards Antoine. From skip at pobox.com Sun Apr 25 23:55:32 2010 From: skip at pobox.com (skip at pobox.com) Date: Sun, 25 Apr 2010 16:55:32 -0500 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> Message-ID: <19412.47700.803086.819640@montanaro.dyndns.org> >> Given that Sean is vouching for him I'm fine with it. Antoine> I'm not sure I agree. Of course it could be argued the risk is Antoine> minimal, but I think it's better if all people go through the Antoine> same path of proving their motivation and quality of work. And Antoine> if there's something wrong with that process we'd better Antoine> address it than give random privileges to people we like :) Let me expand on my original message. Sean has been an integral part of the Python community for many years, keeping much of our hardware and software humming and providing critical network expertise at PyCon. I don't think we have to be such slaves to a set of rules that we can't use an implicit trust network to make decisions in certain cases. I trust Sean's judgement. Skip From benjamin at python.org Sun Apr 25 23:59:14 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 25 Apr 2010 16:59:14 -0500 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <19412.47700.803086.819640@montanaro.dyndns.org> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <19412.47700.803086.819640@montanaro.dyndns.org> Message-ID: 2010/4/25 : > > ? ?>> Given that Sean is vouching for him I'm fine with it. > > ? ?Antoine> I'm not sure I agree. Of course it could be argued the risk is > ? ?Antoine> minimal, but I think it's better if all people go through the > ? ?Antoine> same path of proving their motivation and quality of work. ?And > ? ?Antoine> if there's something wrong with that process we'd better > ? ?Antoine> address it than give random privileges to people we like :) > > Let me expand on my original message. ?Sean has been an integral part of the > Python community for many years, keeping much of our hardware and software > humming and providing critical network expertise at PyCon. ?I don't think we > have to be such slaves to a set of rules that we can't use an implicit trust > network to make decisions in certain cases. ?I trust Sean's judgement. I don't think Antoine is questioning Sean's judgement but rather that we should get into the habit of giving some people "shortcuts" through the regular process. -- Regards, Benjamin From exarkun at twistedmatrix.com Mon Apr 26 00:18:19 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Apr 2010 22:18:19 -0000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> Message-ID: <20100425221819.2083.789388942.divmod.xquotient.14@localhost.localdomain> On 09:39 pm, solipsis at pitrou.net wrote: > pobox.com> writes: >> >> >> Sean> However, I will step up for him and say that I've known him >>a >> Sean> decade, and he's very trustworthy. He has been the >>president (we >> Sean> call that position Maximum Leader) of our Linux Users Group >>here >> Sean> for 5 years or so. >> >>Given that Sean is vouching for him I'm fine with it. > >I'm not sure I agree. Of course it could be argued the risk is minimal, >but I >think it's better if all people go through the same path of proving >their >motivation and quality of work. >And if there's something wrong with that process we'd better address it >than >give random privileges to people we like :) +1 As others have said, I don't have any problem with Sean's judgement. That's not what's being questioned here. Jean-Paul From solipsis at pitrou.net Mon Apr 26 00:18:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Apr 2010 22:18:47 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <19412.47700.803086.819640@montanaro.dyndns.org> Message-ID: Le Sun, 25 Apr 2010 16:59:14 -0500, Benjamin Peterson a ?crit?: > > I don't think Antoine is questioning Sean's judgement but rather that we > should get into the habit of giving some people "shortcuts" through the > regular process. Yes, exactly. If we often take shortcuts with our own process, it can appear unfair and demotivating to regular people (who must go through the normal process). I'm sure we all know people who are demonstrably competent in other communities, still we don't give them privileges right away. From tseaver at palladion.com Mon Apr 26 00:48:03 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 25 Apr 2010 18:48:03 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> Message-ID: <4BD4C6A3.3030409@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: > pobox.com> writes: >> >> Sean> However, I will step up for him and say that I've known him a >> Sean> decade, and he's very trustworthy. He has been the president (we >> Sean> call that position Maximum Leader) of our Linux Users Group here >> Sean> for 5 years or so. >> >> Given that Sean is vouching for him I'm fine with it. > > I'm not sure I agree. Of course it could be argued the risk is minimal, but I > think it's better if all people go through the same path of proving their > motivation and quality of work. > And if there's something wrong with that process we'd better address it than > give random privileges to people we like :) I think there is a definite "unpriced externality" to keeping the process barriers high here. I don't belive from conversations at the language summit / PyCon that the community is being overrun with hordes of unworthies clamoring to triage Python bugs: rather the opposite, in fact. It seems to me that backing from an established community member ought to be enough to get a prospective triageur at least provisional roles to do the work, with the caveat that it might be revoked it it didn't turn out well. If it does turn out well, then look to *expand* that user's roles in the community, with a nice helping of public acclaim to go with it. I am not arguing for "making exceptions for friends" here; rather that the acknowledged issues with inclusiveness / espansion of the developer community require making changes to the rules to encourage more participation. BTW, language like "prov[ing] their motivation" is itself demotivating, and likely contributes to the status quo ante. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvUxqMACgkQ+gerLs4ltQ7Z9gCgn9Rox2dPLR/Vkj9WLMziUdcl 9a8AoLgzXSIUAKsibm1e2ww9feBNd3P/ =iHgN -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Apr 26 00:48:03 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 25 Apr 2010 18:48:03 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> Message-ID: <4BD4C6A3.3030409@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: > pobox.com> writes: >> >> Sean> However, I will step up for him and say that I've known him a >> Sean> decade, and he's very trustworthy. He has been the president (we >> Sean> call that position Maximum Leader) of our Linux Users Group here >> Sean> for 5 years or so. >> >> Given that Sean is vouching for him I'm fine with it. > > I'm not sure I agree. Of course it could be argued the risk is minimal, but I > think it's better if all people go through the same path of proving their > motivation and quality of work. > And if there's something wrong with that process we'd better address it than > give random privileges to people we like :) I think there is a definite "unpriced externality" to keeping the process barriers high here. I don't belive from conversations at the language summit / PyCon that the community is being overrun with hordes of unworthies clamoring to triage Python bugs: rather the opposite, in fact. It seems to me that backing from an established community member ought to be enough to get a prospective triageur at least provisional roles to do the work, with the caveat that it might be revoked it it didn't turn out well. If it does turn out well, then look to *expand* that user's roles in the community, with a nice helping of public acclaim to go with it. I am not arguing for "making exceptions for friends" here; rather that the acknowledged issues with inclusiveness / espansion of the developer community require making changes to the rules to encourage more participation. BTW, language like "prov[ing] their motivation" is itself demotivating, and likely contributes to the status quo ante. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvUxqMACgkQ+gerLs4ltQ7Z9gCgn9Rox2dPLR/Vkj9WLMziUdcl 9a8AoLgzXSIUAKsibm1e2ww9feBNd3P/ =iHgN -----END PGP SIGNATURE----- From steve at holdenweb.com Mon Apr 26 01:18:27 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 25 Apr 2010 19:18:27 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD4C6A3.3030409@palladion.com> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> Message-ID: Tres Seaver wrote: > Antoine Pitrou wrote: >> pobox.com> writes: >>> Sean> However, I will step up for him and say that I've known him a >>> Sean> decade, and he's very trustworthy. He has been the president (we >>> Sean> call that position Maximum Leader) of our Linux Users Group here >>> Sean> for 5 years or so. >>> >>> Given that Sean is vouching for him I'm fine with it. >> I'm not sure I agree. Of course it could be argued the risk is minimal, but I >> think it's better if all people go through the same path of proving their >> motivation and quality of work. >> And if there's something wrong with that process we'd better address it than >> give random privileges to people we like :) > > I think there is a definite "unpriced externality" to keeping the > process barriers high here. I don't belive from conversations at the > language summit / PyCon that the community is being overrun with hordes > of unworthies clamoring to triage Python bugs: rather the opposite, in > fact. It seems to me that backing from an established community member > ought to be enough to get a prospective triageur at least provisional > roles to do the work, with the caveat that it might be revoked it it > didn't turn out well. If it does turn out well, then look to *expand* > that user's roles in the community, with a nice helping of public > acclaim to go with it. > > I am not arguing for "making exceptions for friends" here; rather that > the acknowledged issues with inclusiveness / espansion of the developer > community require making changes to the rules to encourage more > participation. > > BTW, language like "prov[ing] their motivation" is itself demotivating, > and likely contributes to the status quo ante. With my PSF hat on I'd like to support Tres here (and, by extension, Sean's proposal). Lowering the barriers of entry is a desirable goal. If adding people created work for already-busy developers then I'd be against it*, but with Sean offering to mentor his new protege and ensure that he limits his role to triage initially that doesn't seem to be an issue. Maybe it's time to review the way people "prove their motivation and the quality of their work"? regards Steve * I'd be against it, but I'd fight to change the development process so that adding new people *didn't* create work. We should, in my opinion, be looking for a continual influx of new worker bees. -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Mon Apr 26 01:24:08 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 26 Apr 2010 00:24:08 +0100 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> Message-ID: <4BD4CF18.30206@voidspace.org.uk> On 26/04/2010 00:18, Steve Holden wrote: > Tres Seaver wrote: > >> Antoine Pitrou wrote: >> >>> pobox.com> writes: >>> >>>> Sean> However, I will step up for him and say that I've known him a >>>> Sean> decade, and he's very trustworthy. He has been the president (we >>>> Sean> call that position Maximum Leader) of our Linux Users Group here >>>> Sean> for 5 years or so. >>>> >>>> Given that Sean is vouching for him I'm fine with it. >>>> >>> I'm not sure I agree. Of course it could be argued the risk is minimal, but I >>> think it's better if all people go through the same path of proving their >>> motivation and quality of work. >>> And if there's something wrong with that process we'd better address it than >>> give random privileges to people we like :) >>> >> I think there is a definite "unpriced externality" to keeping the >> process barriers high here. I don't belive from conversations at the >> language summit / PyCon that the community is being overrun with hordes >> of unworthies clamoring to triage Python bugs: rather the opposite, in >> fact. It seems to me that backing from an established community member >> ought to be enough to get a prospective triageur at least provisional >> roles to do the work, with the caveat that it might be revoked it it >> didn't turn out well. If it does turn out well, then look to *expand* >> that user's roles in the community, with a nice helping of public >> acclaim to go with it. >> >> I am not arguing for "making exceptions for friends" here; rather that >> the acknowledged issues with inclusiveness / espansion of the developer >> community require making changes to the rules to encourage more >> participation. >> >> BTW, language like "prov[ing] their motivation" is itself demotivating, >> and likely contributes to the status quo ante. >> > With my PSF hat on I'd like to support Tres here (and, by extension, > Sean's proposal). Lowering the barriers of entry is a desirable goal. > > If adding people created work for already-busy developers then I'd be > against it*, but with Sean offering to mentor his new protege and ensure > that he limits his role to triage initially that doesn't seem to be an > issue. > > Maybe it's time to review the way people "prove their motivation and the > quality of their work"? > Perhaps mentoring by an established committer could become a *standard* acceptable way to gain tracker privileges. It makes a lot of sense for the barriers to entry for bug triaging to be substantially lower than for commit privileges. I agree that we should try and establish straightforward and consistent procedures, but also agree that those procedures should serve the community rather than vice-versa. All the best, Michael > regards > Steve > > * I'd be against it, but I'd fight to change the development process so > that adding new people *didn't* create work. We should, in my opinion, > be looking for a continual influx of new worker bees. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From orsenthil at gmail.com Mon Apr 26 02:24:35 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 26 Apr 2010 05:54:35 +0530 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <19412.47700.803086.819640@montanaro.dyndns.org> Message-ID: <20100426002435.GA4022@remy> On Sun, Apr 25, 2010 at 10:18:47PM +0000, Antoine Pitrou wrote: > > I don't think Antoine is questioning Sean's judgement but rather that we > > should get into the habit of giving some people "shortcuts" through the > > regular process. > > Yes, exactly. > If we often take shortcuts with our own process, it can appear unfair and > demotivating to regular people (who must go through the normal process). I agree with Antoine's point here. As much as I respect Sean and his contributions, it is important to consider the implications as it may appear to others. If you look at Daniel Diniz, who has enhanced tracker privileges, he started off by using the normal tracker privilege commenting on bugs, patches and within *weeks*, he started triaging bugs with enhanced privs. That kind of seems to me a middle-way, as in you start off triaging in a normal mode with a backing of mentor, it becomes a easy way to upgrade very soon. -- Senthil Man must shape his tools lest they shape him. -- Arthur R. Miller From tjreedy at udel.edu Mon Apr 26 02:42:00 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Apr 2010 20:42:00 -0400 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: <4BD4A69A.4080507@tummy.com> References: <4BD4A69A.4080507@tummy.com> Message-ID: On 4/25/2010 4:31 PM, Sean Reifschneider wrote: > I'm trying to get a good friend of mine to start doing bug triage on Python. What is *his* interest? How long has he known and used Python? > As part of my trying to mentor him on it, I've found that many of the common > things I do in triage, like setting a priority for priorityless bugs, > assigning them to people who obviously are the next step, requires enhanced > privileges. And enhanced knowledge of the process (and people, for assigning). > He has no reputation in the Python community, so I'd be up for getting him > started on things that require fewer privileges like verifying older patches > still apply against newer Pythons, or maybe summarizing priority/assignment > changes to the list and having someone (possibly me) make the changes, etc... To me, this seems the appropriate way to start. Has he registered for the tracker yet? With his real name? How many issues has he commented on? I searched for issues with 'dangerjim' on the nosy list and found none. I think someone should have read at least, say, 10 issues and commented on at least, say, 5, to show real interest and sanity before being elevated. > However, I will step up for him and say that I've known him a decade, and he's > very trustworthy. He has been the president (we call that position Maximum > Leader) of our Linux Users Group here for 5 years or so. > > Thoughts? I consider particular knowledge and interest to be as important and trustworthness. Terry Jan Reedy From exarkun at twistedmatrix.com Mon Apr 26 04:25:33 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 26 Apr 2010 02:25:33 -0000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> Message-ID: <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> On 25 Apr, 11:18 pm, steve at holdenweb.com wrote: >Tres Seaver wrote: >>Antoine Pitrou wrote: >>> pobox.com> writes: >>>> Sean> However, I will step up for him and say that I've known >>>>him a >>>> Sean> decade, and he's very trustworthy. He has been the >>>>president (we >>>> Sean> call that position Maximum Leader) of our Linux Users >>>>Group here >>>> Sean> for 5 years or so. >>>> >>>>Given that Sean is vouching for him I'm fine with it. >>>I'm not sure I agree. Of course it could be argued the risk is >>>minimal, but I >>>think it's better if all people go through the same path of proving >>>their >>>motivation and quality of work. >>>And if there's something wrong with that process we'd better address >>>it than ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Don't overlook this part of Antoine's post. >>>give random privileges to people we like :) >> >>I think there is a definite "unpriced externality" to keeping the >>process barriers high here. I don't belive from conversations at the >>language summit / PyCon that the community is being overrun with >>hordes >>of unworthies clamoring to triage Python bugs: rather the opposite, >>in >>fact. It seems to me that backing from an established community >>member >>ought to be enough to get a prospective triageur at least provisional >>roles to do the work, with the caveat that it might be revoked it it >>didn't turn out well. If it does turn out well, then look to *expand* >>that user's roles in the community, with a nice helping of public >>acclaim to go with it. >> >>I am not arguing for "making exceptions for friends" here; rather >>that >>the acknowledged issues with inclusiveness / espansion of the >>developer >>community require making changes to the rules to encourage more >>participation. >> >>BTW, language like "prov[ing] their motivation" is itself >>demotivating, >>and likely contributes to the status quo ante. > >With my PSF hat on I'd like to support Tres here (and, by extension, >Sean's proposal). Lowering the barriers of entry is a desirable goal. > >If adding people created work for already-busy developers then I'd be >against it*, but with Sean offering to mentor his new protege and >ensure >that he limits his role to triage initially that doesn't seem to be an >issue. > >Maybe it's time to review the way people "prove their motivation and >the >quality of their work"? Sounds good. Why is the barrier for this permission any higher than someone asking for it? Is there really a need to protect against contributors with malicious intent? I think there should be a page on python.org that says all contributors are welcome, and one way to become a contributor is to wrangle the issue tracker, and explains what this involves (I don't really have any idea, actually; I assume it's things like setting the owner of new tickets to someone who might actually fix it, things that would happen automatically if roundup had the right information), and then anyone who steps up gets the necessary access. Jean-Paul >regards >Steve > >* I'd be against it, but I'd fight to change the development process so >that adding new people *didn't* create work. We should, in my opinion, >be looking for a continual influx of new worker bees. >-- >Steve Holden +1 571 484 6266 +1 800 494 3119 From stephen at xemacs.org Mon Apr 26 05:48:10 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 26 Apr 2010 12:48:10 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD4C6A3.3030409@palladion.com> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> Message-ID: <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> Tres Seaver writes: > I think there is a definite "unpriced externality" to keeping the > process barriers high here. The proposed trial period is not a high barrier, except to those who really didn't want to being doing the work anyway. Note that There is also an externality to having accounts with unused privileges lying around the server. The fact is that there just aren't very many people willing to do the work. I've considered doing it (especially since Daniel has gone out of his way to inform me of fixes to the Python tracker, since I maintain a couple of Roundup instances for other projects), but I just don't have the time. Both personally and in general, I really don't think this barrier is high enough that removal would make a significant difference. The example of Daniel Diniz *is* salient. A couple of weeks of doing things the long way (which forces the mentor to look at them, among other things), a week for discussion by the people who work most closely with the candidate and the administrators of the host/tracker where privileges will be granted -- if this matters in the grand scheme of things, the person probably didn't want to be doing the work in the first place. They should find a way to contribute more suited to their capabilities and interests. > I am not arguing for "making exceptions for friends" here; rather that > the acknowledged issues with inclusiveness / espansion of the developer > community require making changes to the rules to encourage more > participation. > > BTW, language like "prov[ing] their motivation" is itself demotivating, > and likely contributes to the status quo ante. Unlikely IMHO. Yes, if you allow anybody to make changes, you'll get more contributions. You'll also create a lot of janitorial work for the people who take care of project hygiene, and *they* will lose incentive. That is a *really* big risk. Remember, because of the sexiness/mission criticality of the Linux kernel, a couple of *hundred* people world wide get paid to spend significant amounts of time on it. That's not the comparison to make here; Python is a great language, but there are several reasonable alternatives, and it's not suited to all projects. Look at the 99.5% of Sourceforge projects (or the fact that GNU Savannah can't attract contributions to get the "official" GNU VCS -- Bazaar -- working properly), and you'll see that Python actually has a really well- functioning, attractive process. From regebro at gmail.com Mon Apr 26 07:48:20 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Apr 2010 07:48:20 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: I'd say there is something wrong with the process. If a trusted developer can't get somebody more privilege on the tracker by saying that "I trust this guy", then a new process is needed. That's it's too hard to get privileges in the Python development community has been evident too long, I think. There is one privilege that should be hard to get: Permanent delete. But being able to triage bugs isn't such a privilege. Heck, not even commit access is, because of someone makes something bad, you can back out the checkin. Giving people rights to a bugtracker or versioning system is not dangerous, and should not be hard. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From jafo at tummy.com Mon Apr 26 08:12:23 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Mon, 26 Apr 2010 00:12:23 -0600 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> Message-ID: <20100426061223.GA17467@tummy.com> On Sun, Apr 25, 2010 at 08:42:00PM -0400, Terry Reedy wrote: >What is *his* interest? How long has he known and used Python? Good points have been made on both sides of the issue here. Despite my having a vested interest, I really have no strong feelings one way or another on the initial request. But, the answers to your questions above may make it more clear why I was looking for the enhanced privileges. James (dangerjim) has *NO* knowledge of Python -- he has done primarily C and Java coding. He *DOES* have time on his hands. This is why I proposed to him to do tracker triage. However, as I started walking him through some examples of triage, I realized that regular accounts don't have the privileges to do any of the things I was proposing. For example: Go into the list of task tasks and look at the ones with no priority. Read through the description and any follow-ups and try to figure out what priority to give it. In most cases it will be "normal". However, for some issues it will be clear they should be a higher or lower priority, even to someone who doesn't know Python. Then we went on to issue 5575 and read through it. In reading this one to determine the priority, it was clear that the ball was back in Collin's court, so I showed that I would look to see if Collin was a valid assignee (which he was) and assign it to him, with a comment about why. Go into old bugs that have patches, and see if the patches cleanly apply against the trunk. If they do, do a "make" and "make test". Add a comment with the outcome. Two of the 3 easiest things I came up with for an outsider to help out with, are things that his account couldn't do. But, as I said above, I'm fine with having him push those changes through me and I can make them when he can't. Thanks, Sean -- "It was a sneaky lawyer's trick, and I fell for it. Because... She's much smarter than me." -- _High_Fidelity_ Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From martin at v.loewis.de Mon Apr 26 08:26:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Apr 2010 08:26:21 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> Message-ID: <4BD5320D.5030502@v.loewis.de> > If adding people created work for already-busy developers then I'd be > against it* I most certainly does create work, but that could be as little as sending an email message to some administrator. There is no other way: somebody will have to make a decision, and that is "work". Regards, Martin From martin at v.loewis.de Mon Apr 26 08:33:04 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Apr 2010 08:33:04 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> Message-ID: <4BD533A0.1000802@v.loewis.de> > Sounds good. Why is the barrier for this permission any higher than > someone asking for it? Is there really a need to protect against > contributors with malicious intent? There is a little risk. People doing triage can make two common mistakes, and both do happen in the Python tracker from time to time: a) they reject some contribution, even though a long-time contributor might have accepted it with modifications; sometimes this rejection happens for overly formal reasons, and b) they assign issues to someone who might be formally in charge, but is unlikely to act on the issue in a reasonable amount of time. Of course, long-term contributors can and do also make the same mistakes; it's just that new people are often unfamiliar with the conventions. This was all in the abstract, independent of dangerjim (whom I don't know). Regards, Martin From regebro at gmail.com Mon Apr 26 09:14:40 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Apr 2010 09:14:40 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD533A0.1000802@v.loewis.de> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> <4BD533A0.1000802@v.loewis.de> Message-ID: On Mon, Apr 26, 2010 at 08:33, "Martin v. L?wis" wrote: > There is a little risk. People doing triage can make two common > mistakes, and both do happen in the Python tracker from time to time: > a) they reject some contribution, even though a long-time contributor > ? might have accepted it with modifications; sometimes this rejection > ? happens for overly formal reasons, and > b) they assign issues to someone who might be formally in charge, but > ? is unlikely to act on the issue in a reasonable amount of time. Sure. But these errors can be fixed, just as a checkin can be reverted. That's what I mean with the risk being low, you can't make permanent damage. The Zope community gives commit access by recommendation. This works well. The easier it is to contribute, the more people contributes. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From p.f.moore at gmail.com Mon Apr 26 10:19:10 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Apr 2010 09:19:10 +0100 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: <20100426061223.GA17467@tummy.com> References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: On 26 April 2010 07:12, Sean Reifschneider wrote: > Two of the 3 easiest things I came up with for an outsider to help out > with, are things that his account couldn't do. Would in not therefore be reasonable to question whether the *default* privileges be changed to allow "outsiders" to do these things, rather than asking for escalated privileges for one individual? (Or more likely, as well as doing so, as changing the default is likely to be a longer discussion...) Paul. From asmodai at in-nomine.org Mon Apr 26 11:31:53 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Mon, 26 Apr 2010 11:31:53 +0200 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: <20100426093153.GA61243@nexus.in-nomine.org> -On [20100426 10:21], Paul Moore (p.f.moore at gmail.com) wrote: >Would in not therefore be reasonable to question whether the *default* >privileges be changed to allow "outsiders" to do these things, rather >than asking for escalated privileges for one individual? (Or more >likely, as well as doing so, as changing the default is likely to be a >longer discussion...) Be careful. Trackers are often hit by spam bots which change random form field values. I cannot from memory recall whether or not we had this issue on either the tracker tracker or Python's tracker. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B A wise man that walks in the dark with a blindfold on, is not much of a wise man... From ziade.tarek at gmail.com Mon Apr 26 12:04:33 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 26 Apr 2010 12:04:33 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <20100422160154.803693A4070@sparrow.telecommunity.com> Message-ID: On Thu, Apr 22, 2010 at 6:05 PM, Tarek Ziad? wrote: > 2010/4/22 P.J. Eby : >> At 10:54 AM 4/22/2010 +0200, Tarek Ziad? wrote: >>> >>> Hello, >>> >>> I think I went through all the latest feedback regarding PEP 376. >>> >>> There will be still some work of course, on the implementation side >>> (for instance the Zip issues described by PJE). >>> >>> But I would like to go ahead and propose PEP 376 for acceptance. >> >> One final (optional) question/suggestion... ?should we change from this (in >> RECORD files): >> >> ? ?lib/python2.6/site-packages/docutils/__init__.pyc >> >> to this: >> >> ? ?lib/python2.6/site-packages/docutils/__init__.pyc,, >> >> In this way, reader code can be written as: >> >> ? ?for path, hash, size in csv.reader(...): >> >> It's not a high-priority thing, but if anybody is writing code to parse >> RECORD files outside the provided API implementation (eg. system packaging >> tool authors), they'll probably appreciate this. ?;-) >> > > Yes, of course. the RECORD file is supposed to be iterable using the csv reader, > so that's a mistake in the PEP. > > Thanks for noticing, I'll fix it later today This is fixed. Guido told me in private he's now accepting the PEP since there's consensus. The next step is to finish PEP 376 implementation in distutils2, and pkgutil. There will be a full backport of the new pkgutil in distutils2, so third party projects that wants to be compatible with this new standard will be able to start doing it before Python 3.2 is out. Thanks to all the folks that helped in this process ! Regards Tarek -- Tarek Ziad? | http://ziade.org From stephen at xemacs.org Mon Apr 26 12:58:20 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 26 Apr 2010 19:58:20 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > I'd say there is something wrong with the process. If a trusted > developer can't get somebody more privilege on the tracker by > saying that "I trust this guy", then a new process is needed. > That's it's too hard to get privileges in the Python development > community has been evident too long, I think. It is entirely *not* evident to me that it's too hard to get privileges in the Python development community (Python's development process works -- and it works really well by comparison to 99% of the processes out there). And processes are delicate; they should be changed only when the people involved in them have the time and inclination to work on rebalancing them. > There is one privilege that should be hard to get: Permanent delete. > But being able to triage bugs isn't such a privilege. Heck, not even > commit access is, because of someone makes something bad, you can back > out the checkin. Sure, but that's still *work*, and it's work for *somebody else*. The person who made the mistake is unlikely to detect it, and needs to be told to fix it, if they even fix it themselves. > Giving people rights to a bugtracker or versioning system is not > dangerous, and should not be hard. As someone who does a lot more managing of shared resources than coding in the projects I'm active in, I disagree about the danger. Enthusiastic newbies can do a lot of minor damage in a short period of time, and cleaning that up is *work*. This danger is almost entirely mitigated by a small amount of mentoring -- which is precisely what the current process requires -- not only of the recomending party, but also of the existing workers. I'm not claiming that the current balance is right. Just that it's not obvious that it's *wrong*, and therefore the decision should be left up to the people who will do the mentoring, the supervision, and -- if necessary -- the cleanup. If the existing tracker crew is happy with Sean's recommendation, and similar recommendations in the future, I'm happy too. But it is a process change, and they should be comfortable with it. From fuzzyman at voidspace.org.uk Mon Apr 26 13:05:01 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 26 Apr 2010 12:05:01 +0100 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BD5735D.7010202@voidspace.org.uk> On 26/04/2010 11:58, Stephen J. Turnbull wrote: > [snip...] > > I'm not claiming that the current balance is right. Hmm... the core development team (those who make commits once a month or more frequently) is very small, the number of people doing bug triaging is currently good but also small. We have patches and issues in the tracker that may have responses but will never get properly looked at because no-one on the core team is interested or has the mental bandwidth, we have possibly hundreds of modules in the standard library without a maintainer. I think it is very much in the interest of Python to evolve our processes in order to encourage more core-developers. Evolving means experimenting *and* being willing to change. It is certainly less *effort* to accept the status quo, but with several more committed and enthusiastic (and good) core developers there is an awful lot (more) we could achieve. All the best, Michael Foord > Just that it's > not obvious that it's *wrong*, and therefore the decision should be > left up to the people who will do the mentoring, the supervision, and > -- if necessary -- the cleanup. If the existing tracker crew is happy > with Sean's recommendation, and similar recommendations in the future, > I'm happy too. But it is a process change, and they should be > comfortable with it. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From solipsis at pitrou.net Mon Apr 26 13:24:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 13:24:47 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD5735D.7010202@voidspace.org.uk> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> Message-ID: <1272281087.3539.22.camel@localhost> Hello, > I think it is very much in the interest of Python to evolve our > processes in order to encourage more core-developers. Evolving means > experimenting *and* being willing to change. It is certainly less > *effort* to accept the status quo, but with several more committed and > enthusiastic (and good) core developers there is an awful lot (more) we > could achieve. I certainly agree we should try to attract more good-willed and competent contributors. I also agree with Stephen that, in a project with a non-trivial amount of complexity such as CPython, not having (tracker or commit) privileges is not the real barrier to entry. You have to learn how the software works, how its development works, who are the people working on it, etc. Regards Antoine. From ncoghlan at gmail.com Mon Apr 26 13:30:32 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Apr 2010 21:30:32 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BD57958.5050002@gmail.com> Stephen J. Turnbull wrote: > > There is one privilege that should be hard to get: Permanent delete. > > But being able to triage bugs isn't such a privilege. Heck, not even > > commit access is, because of someone makes something bad, you can back > > out the checkin. > > Sure, but that's still *work*, and it's work for *somebody else*. The > person who made the mistake is unlikely to detect it, and needs to be > told to fix it, if they even fix it themselves. As someone who's main contribution these days (other than kibbitzing here) is reviewing checkins that go by on python-checkins, there generally *is* a slight uptick in that workload when someone new is given commit privileges (e.g. reminders to update docs, NEWS, tests, what's new, ACKS, pointing out potential cross-platform or backwards compatibility problems, reminders to check with active maintainers for some modules). It usually doesn't last long, because those people are currently familiar with the basics of the process from submitting patches to the tracker and seeing them applied for a while before commit privileges are granted. Actually having to revert commits and rerun the test suite to make sure everything is back the way it should be can probably be taken as a 1-for-1 loss in patches applied without being too far off the mark (fortunately, that is rather rare under the current system - I expect it would be significantly more common if we were more casual about handing out commit privileges). However, I will also point out that a large chunk of the motivation in moving to a DVCS is to make life easier for *non-committers*. Existing devs get some benefit in being able to use something less clunky than svnmerge to manage the maintenance branches, but that is pretty minor compared to the gain in usability for everyone else. > If the existing tracker crew is happy > with Sean's recommendation, and similar recommendations in the future, > I'm happy too. But it is a process change, and they should be > comfortable with it. Agreed that it is the existing triage team that really needs to OK this (since they'll likely be the ones cleaning up any mistakes). Getting someone to do things the clunky way for a couple of weeks still seems like a decent way for them to show they understand the basics of our tracker management policies (since merely commenting on issues can't cause a mess the way inadvertent misuse of tracker privileges can). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From scott+python-dev at scottdial.com Mon Apr 26 13:40:31 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 26 Apr 2010 07:40:31 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <1272281087.3539.22.camel@localhost> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> Message-ID: <4BD57BAF.9040808@scottdial.com> On 4/26/2010 7:24 AM, Antoine Pitrou wrote: >> I think it is very much in the interest of Python to evolve our >> processes in order to encourage more core-developers. Evolving means >> experimenting *and* being willing to change. It is certainly less >> *effort* to accept the status quo, but with several more committed and >> enthusiastic (and good) core developers there is an awful lot (more) we >> could achieve. > > I certainly agree we should try to attract more good-willed and > competent contributors. > > I also agree with Stephen that, in a project with a non-trivial amount > of complexity such as CPython, not having (tracker or commit) privileges > is not the real barrier to entry. You have to learn how the software > works, how its development works, who are the people working on it, etc. > I'd like to respond to Michael's comment about the "possibly hundreds of modules in the standard library without a maintainer." My own experience (issue5949) has been positive despite the lack of a dedicated maintainer. When I had my own itch to scratch, nobody stopped me from scratching it.. some people told me when I could scratch it and how they'd like it scratched.. but I wasn't ignored or rejected despite the lack of a maintainer. Thanks to RDM for giving my issue attention. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From steve at holdenweb.com Mon Apr 26 13:40:10 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 26 Apr 2010 07:40:10 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD5320D.5030502@v.loewis.de> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <4BD5320D.5030502@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> If adding people created work for already-busy developers then I'd be >> against it* > > I most certainly does create work, but that could be as little as > sending an email message to some administrator. > > There is no other way: somebody will have to make a decision, and that > is "work". > Well, I'm sorry to have put you to the work of penning that reply, when you could have used the effort instead to triage a bug. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Mon Apr 26 13:43:28 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 26 Apr 2010 12:43:28 +0100 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD57BAF.9040808@scottdial.com> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57BAF.9040808@scottdial.com> Message-ID: <4BD57C60.4080307@voidspace.org.uk> On 26/04/2010 12:40, Scott Dial wrote: > On 4/26/2010 7:24 AM, Antoine Pitrou wrote: > >>> I think it is very much in the interest of Python to evolve our >>> processes in order to encourage more core-developers. Evolving means >>> experimenting *and* being willing to change. It is certainly less >>> *effort* to accept the status quo, but with several more committed and >>> enthusiastic (and good) core developers there is an awful lot (more) we >>> could achieve. >>> >> I certainly agree we should try to attract more good-willed and >> competent contributors. >> >> I also agree with Stephen that, in a project with a non-trivial amount >> of complexity such as CPython, not having (tracker or commit) privileges >> is not the real barrier to entry. You have to learn how the software >> works, how its development works, who are the people working on it, etc. >> >> > I'd like to respond to Michael's comment about the "possibly hundreds of > modules in the standard library without a maintainer." My own experience > (issue5949) has been positive despite the lack of a dedicated > maintainer. When I had my own itch to scratch, nobody stopped me from > scratching it.. some people told me when I could scratch it and how > they'd like it scratched.. but I wasn't ignored or rejected despite the > lack of a maintainer. Thanks to RDM for giving my issue attention. > > Right, but finding counterexamples is not at all hard... Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Mon Apr 26 13:45:34 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 26 Apr 2010 12:45:34 +0100 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <1272281087.3539.22.camel@localhost> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> Message-ID: <4BD57CDE.7080705@voidspace.org.uk> On 26/04/2010 12:24, Antoine Pitrou wrote: > Hello, > > >> I think it is very much in the interest of Python to evolve our >> processes in order to encourage more core-developers. Evolving means >> experimenting *and* being willing to change. It is certainly less >> *effort* to accept the status quo, but with several more committed and >> enthusiastic (and good) core developers there is an awful lot (more) we >> could achieve. >> > I certainly agree we should try to attract more good-willed and > competent contributors. > > I also agree with Stephen that, in a project with a non-trivial amount > of complexity such as CPython, not having (tracker or commit) privileges > is not the real barrier to entry. You have to learn how the software > works, how its development works, who are the people working on it, etc. > > So the question remains - for *tracker* privileges, should the recommendation and commitment to mentor from an established commiter be sufficient (and therefore a standard part of our process)? I think this is a reasonable barrier for entry and should be acceptable. It should be stated as part of our standard procedure however rather than being an exception to our standard procedure. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ndbecker2 at gmail.com Mon Apr 26 14:20:10 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 26 Apr 2010 08:20:10 -0400 Subject: [Python-Dev] argparse suggestion Message-ID: steven.bethard at gmail.com made a very nice module for me to enhance argparse called argparse_bool.py, which contains ConfigureAction. This will allow a boolean value to be set very like the gnu configure style: --foo --with-foo --without-foo --no-foo --foo=yes --foo=no I've been happily using it, and I think it would be of sufficient general interest to include it with the standard library. -------------- next part -------------- A non-text attachment was scrubbed... Name: argparse_bool.py Type: text/x-python Size: 2366 bytes Desc: not available URL: From amk at amk.ca Mon Apr 26 14:14:17 2010 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 26 Apr 2010 08:14:17 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <20100426022533.2083.1879382062.divmod.xquotient.57@localhost.localdomain> Message-ID: <20100426121417.GA2448@amk-desktop.matrixgroup.net> On Mon, Apr 26, 2010 at 02:25:33AM -0000, exarkun at twistedmatrix.com wrote: > I think there should be a page on python.org that says all > contributors are welcome, and one way to become a contributor is to > wrangle the issue tracker, and explains what this involves (I don't > really have any idea, actually; I assume it's things like setting Actually there already is such a page: http://www.python.org/dev/contributing/ . Excerpt: If you have helped out in the issue tracker for a little while or have been a good participant on python-dev you may ask for Developer privileges on the tracker which allow you to change any and all metadata on an issue. Please don't be shy about asking for the privilege! We are more liberal with giving out this ability than with commit privileges, so you don't need to have been contributing for a year to gain this ability. And with Developer privileges you can work more autonomously and be an even greater help by narrowing down what issues on the tracker deserve the most attention at any one time. Issue workflow is specified in http://www.python.org/dev/workflow/. Suggestions for text improvements/changes are welcomed. --amk From orsenthil at gmail.com Mon Apr 26 14:27:42 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 26 Apr 2010 17:57:42 +0530 Subject: [Python-Dev] argparse suggestion In-Reply-To: References: Message-ID: <20100426122742.GD9423@remy> On Mon, Apr 26, 2010 at 08:20:10AM -0400, Neal Becker wrote: > steven.bethard at gmail.com made a very nice module for me to enhance argparse > called argparse_bool.py, which contains ConfigureAction. This will allow a Would not it be a feature enhancement request against argparse.py itself? In that case, you might just file a bug against argparse at bugs.python.org. -- Senthil BOFH excuse #138: BNC (brain not connected) From solipsis at pitrou.net Mon Apr 26 14:34:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 12:34:16 +0000 (UTC) Subject: [Python-Dev] argparse suggestion References: <20100426122742.GD9423@remy> Message-ID: Senthil Kumaran gmail.com> writes: > > On Mon, Apr 26, 2010 at 08:20:10AM -0400, Neal Becker wrote: > > steven.bethard gmail.com made a very nice module for me to enhance argparse > > called argparse_bool.py, which contains ConfigureAction. This will allow a > > Would not it be a feature enhancement request against argparse.py > itself? In that case, you might just file a bug against argparse at > bugs.python.org. I would argue that since argparse is now in the standard library, it should be primarily maintained on {bugs, svn}.python.org. We have enough synchronization problems with externally maintained modules. Regards Antoine. From regebro at gmail.com Mon Apr 26 15:34:09 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Apr 2010 15:34:09 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Apr 26, 2010 at 12:58, Stephen J. Turnbull wrote: > It is entirely *not* evident to me that it's too hard to get > privileges in the Python development community (Python's development > process works -- and it works really well by comparison to 99% of the > processes out there). Well, that's true, all to often a project is controlled by a few developers with no intent of sharing access ever. > Sure, but that's still *work*, and it's work for *somebody else*. Yes, but only when the checkin was wrong. For all other checkins, it's *less* work. Hence, a committer needs to basically fudge up every second checkin to cause more work than he relieves work. :) > As someone who does a lot more managing of shared resources than > coding in the projects I'm active in, I disagree about the danger. > Enthusiastic newbies can do a lot of minor damage in a short period of > time, and cleaning that up is *work*. ?This danger is almost entirely > mitigated by a small amount of mentoring -- which is precisely what > the current process requires -- not only of the recomending party, but > also of the existing workers. And I'm saying that the recommending party is enough. If an established developer says "this guy will not fuck things up", then that is enough guarantee that he won't fuck things up. > But it is a process change, and they should be > comfortable with it. Of course. I'm just arguing for that this process change is correct, and that the current balance is wrong, and that a change is comfortable and safe. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From jnoller at gmail.com Mon Apr 26 16:17:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 26 Apr 2010 10:17:18 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD5735D.7010202@voidspace.org.uk> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> Message-ID: On Mon, Apr 26, 2010 at 7:05 AM, Michael Foord wrote: > On 26/04/2010 11:58, Stephen J. Turnbull wrote: >> >> [snip...] >> >> I'm not claiming that the current balance is right. > > Hmm... the core development team (those who make commits once a month or > more frequently) is very small, the number of people doing bug triaging is > currently good but also small. We have patches and issues in the tracker > that may have responses but will never get properly looked at because no-one > on the core team is interested or has the mental bandwidth, we have possibly > hundreds of modules in the standard library without a maintainer. > > I think it is very much in the interest of Python to evolve our processes in > order to encourage more core-developers. Evolving means experimenting *and* > being willing to change. It is certainly less *effort* to accept the status > quo, but with several more committed and enthusiastic (and good) core > developers there is an awful lot (more) we could achieve. > [snip] Just to add fuel to the fire w.r.t this discussion about process-improvements, lowering friction, etc. I'd like to point out (unintentionally tooting my own horn) a discussion I started up on this exact topic last week: http://jessenoller.com/2010/04/22/why-arent-you-contributing-to-python/ http://www.reddit.com/r/Python/comments/burio/why_arent_you_contributing_to_python/ http://news.ycombinator.com/item?id=1285897 I'm going to avoid summarizing the comments - ignoring my original post, I can honestly say I received an alarming amount of feedback some of which was private, but most of which is sitting there for us to possible consume and act on. jesse From alexander.belopolsky at gmail.com Mon Apr 26 17:09:43 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 26 Apr 2010 11:09:43 -0400 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: <20100426093153.GA61243@nexus.in-nomine.org> References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> Message-ID: On Mon, Apr 26, 2010 at 5:31 AM, Jeroen Ruigrok van der Werven wrote: .. > Be careful. Trackers are often hit by spam bots which change random form > field values. I cannot from memory recall whether or not we had this issue > on either the tracker tracker or Python's tracker. > I would think a comment area is a more attractive target for spam bots than "random form field values" such as "assignee" or "priority." (Note that assignee values that a default user can set may be limited to themselves and previous assignees. This will reasonably protect this field from abuse and allow better tracking of whose court the ball is in.) Among the form fields, I would think "nosy" list would be targeted the most because it allows effectively turning the tracker into a mail relay, but I don't remember ever seeing it abused. I also I don't remember ever seeing spam in the bugs.python.org comments which suggests that the subscription process weeds bots reasonably well. +1 on giving dangerjim requested privileges and using his experience to review and change the default privileges. From barry at python.org Mon Apr 26 17:40:05 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Apr 2010 11:40:05 -0400 Subject: [Python-Dev] argparse suggestion In-Reply-To: References: Message-ID: <20100426114005.6a72b97b@heresy> On Apr 26, 2010, at 08:20 AM, Neal Becker wrote: >steven.bethard at gmail.com made a very nice module for me to enhance argparse >called argparse_bool.py, which contains ConfigureAction. This will allow a >boolean value to be set very like the gnu configure style: > >--foo >--with-foo >--without-foo >--no-foo >--foo=yes >--foo=no > >I've been happily using it, and I think it would be of sufficient general >interest to include it with the standard library. +1. This would be a very nice addition to argparse in the stdlib. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From rdmurray at bitdance.com Mon Apr 26 17:42:10 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 26 Apr 2010 11:42:10 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD57CDE.7080705@voidspace.org.uk> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> Message-ID: <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> On Mon, 26 Apr 2010 12:45:34 +0100, Michael Foord wrote: > So the question remains - for *tracker* privileges, should the > recommendation and commitment to mentor from an established commiter be > sufficient (and therefore a standard part of our process)? I think that in a technical sense a commitment to mentoring by an established contributor would be enough. But it seems to me that there are a couple of arguments against it being sufficient in the wider picture. The first is that open source projects tend to be meritocracies. An otherwise unknown person being introduced to the community and immediately given privileges *just* because of the recommendation of another person may feel (especially to the non privileged) like a kind of nepotism. ("It's not what you contribute, it's who you know"). The second is in some ways a subtle variation on the first. If a new person, even with a well respected mentor standing behind them, first approaches the tracker by reviewing and commenting without privileges, it does two things: it allows people in the community who are not the mentor to get a sense of them, and it gives them the benefit of input from people other than the mentor, and all of this happens *before* they have the opportunity (and the worry) of making mistakes(*). Both of these things serve to build community, and the second, IMO, results in a stronger, more confident contributor. I think that someone who has a mentor sponsoring them from the first should be able to go from zero to privileged in a very short period of time (a couple weeks perhaps, mostly depending on their activity level). Someone without a pre-existing mentor could do the same, if their activity level is high enough, and would probably pick up a mentor along the way...or be mentored by #python-dev as a whole if they hang out there. In other words, I think the goal is not just to add new developers to the community, but to continue to build a strong community of developers. -- R. David Murray www.bitdance.com (*) Even a seasoned developer from another project will make mistakes because some of our development process is a part of our culture and not written down, and even that which is written down is not necessarily easy for a newcomer to absorb by reading. From steve at pearwood.info Mon Apr 26 19:30:21 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Apr 2010 03:30:21 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: <201004270330.22604.steve@pearwood.info> On Tue, 27 Apr 2010 01:42:10 am R. David Murray wrote: > On Mon, 26 Apr 2010 12:45:34 +0100, Michael Foord wrote: > > So the question remains - for *tracker* privileges, should the > > recommendation and commitment to mentor from an established > > commiter be sufficient (and therefore a standard part of our > > process)? > > I think that in a technical sense a commitment to mentoring by an > established contributor would be enough. But it seems to me that > there are a couple of arguments against it being sufficient in the > wider picture. > > The first is that open source projects tend to be meritocracies. > An otherwise unknown person being introduced to the community and > immediately given privileges *just* because of the recommendation of > another person may feel (especially to the non privileged) like a > kind of nepotism. ("It's not what you contribute, it's who you > know"). Hang on, are we talking about paying these people? Giving them keys to the executive washroom? Flying them around the world on First Class flights on expensive junkets? Fame and fortune? What privileges are we giving them? Oh yeah, the privilege of working for nothing in a thankless job that takes time and effort and returns nothing but a bullet point on your resume and the knowledge that you've given back to an Open Source project. Who are we worried about offending? The crowds on the Internet who never volunteer for anything, who never submit patches, let alone offer to do the unglamourous work? I think we worry too much about them -- they complain about everything, and have the attention span of a gnat. (Yes, I have had a bad day, why do you ask? *wink*) Other committers? I would hope that, being members of good standing in a meritocracy, they can recognise the difference between "I want my mate to be given triaging privileges because he's my mate", and "I want him to have triaging privileges because I trust him to do a good job". As I see it, the only people who might have a valid reason to be offended or annoyed are those who have volunteered but were rejected for some reason --perhaps because their skills aren't good enough, perhaps because nobody could vouch that their skills are good enough. Well, life is hard, get over it. In a meritocracy it isn't enough to be good at what you do, you also have to be known to be good. DangerJim has (apparently -- I don't know him myself) spent years building *both* his technical skills and his reputation. Sean is willing to vouch for him and mentor him, and if Sean himself has a sufficiently high reputation that Python-Dev is willing to trust his judgement, then I can't see any reason to reject the offer. What's the worst that could happen? He'll do a bad job of triaging bugs, other committers will have to step in and fix it, and Sean will be embarrassed. Nothing irreparable, except possibly Sean's reputation as a judge of others. I don't see this as a high risk move: we're not making him BDFL on Sean's say-so. > The second is in some ways a subtle variation on the first. If a new > person, even with a well respected mentor standing behind them, first > approaches the tracker by reviewing and commenting without > privileges, it does two things: it allows people in the community who > are not the mentor to get a sense of them, Yes, they will have the opportunity, but will they take it? I've submitted two recent patches which have apparently been swallowed by the black hole of "too much to do, too little time to do it". Am I bitter? No, of course not -- I understand that the committers have much to do, I'm a comparative unknown, and while the patches scratch my itch, they obviously don't scratch anyone else's. But this does demonstrate that just because people have the opportunity to "get a sense of them", doesn't mean that they actually will do so -- if anything, the opposite is the case. By increasing the number of untrusted contributors relative to trusted committers, you increase the burden on committers and lower the chances that they will actually review patches. This in turn discourages people from contributing, and the cycle continues. Instead of this vicious circle, I believe that fast-tracking people on the strength of recommendations from *trusted* members is a good way of getting a virtuous circle: more people available to review patches, which means more competent contributors will gain enough of a reputation to be given committer privileges. > and it gives them the > benefit of input from people other than the mentor, and all of this > happens *before* they have the opportunity (and the worry) of making > mistakes(*). Both of these things serve to build community, and the > second, IMO, results in a stronger, more confident contributor. Surely this will depend on the personality of the contributor? Not everyone appreciates being examined like that, even in an informal ad-hoc way, and while they might suck it up and accept it, they don't necessarily benefit from it. I reckon that for every one or two would-be contributors who value having that early oversight, you probably alienate a third enough that he goes elsewhere. -- Steven D'Aprano From tjreedy at udel.edu Mon Apr 26 19:31:35 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 13:31:35 -0400 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> Message-ID: On 4/26/2010 11:09 AM, Alexander Belopolsky wrote: > > I also I don't remember ever seeing spam in the bugs.python.org > comments which suggests that the subscription process weeds bots > reasonably well. And when it fails, spam is deleted just so no one does see it. From eric at trueblade.com Mon Apr 26 19:39:15 2010 From: eric at trueblade.com (Eric Smith) Date: Mon, 26 Apr 2010 13:39:15 -0400 Subject: [Python-Dev] argparse suggestion In-Reply-To: References: Message-ID: <4BD5CFC3.3020600@trueblade.com> Sounds good to me (subject to arguing about spellings, case insensitivity, etc.). Just so it doesn't get lost, I created issue 8538 to track it. Neal Becker wrote: > steven.bethard at gmail.com made a very nice module for me to enhance argparse > called argparse_bool.py, which contains ConfigureAction. This will allow a > boolean value to be set very like the gnu configure style: > > --foo > --with-foo > --without-foo > --no-foo > --foo=yes > --foo=no > > I've been happily using it, and I think it would be of sufficient general > interest to include it with the standard library. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com -- Eric. From tjreedy at udel.edu Mon Apr 26 19:44:55 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 13:44:55 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD57CDE.7080705@voidspace.org.uk> References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> Message-ID: On 4/26/2010 7:45 AM, Michael Foord wrote: > So the question remains - for *tracker* privileges, should the > recommendation and commitment to mentor from an established commiter be > sufficient (and therefore a standard part of our process)? > > I think this is a reasonable barrier for entry and should be acceptable. > It should be stated as part of our standard procedure however rather > than being an exception to our standard procedure. Asking that one make public comments on a minimum of, say, 5 issues is hardly onerous. Terry Jan Reedy From solipsis at pitrou.net Mon Apr 26 19:45:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 17:45:39 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> Message-ID: Steven D'Aprano pearwood.info> writes: > Who are we worried about offending? The crowds on the Internet who never > volunteer for anything, who never submit patches, let alone offer to do > the unglamourous work? Perhaps you should look more carefully. We do have contributors who submit patches and advice on the tracker. There isn't just the committers and the passive masses. (oh, and following your logic, we should ignore your advice, unless you actually contribute to the "unglamourous work" - do you?) > In a meritocracy it isn't enough to be > good at what you do, you also have to be known to be good. If this were the criterion then the answer would be simple: nobody seems to knows dangerjim in the Python community. (to make it clear: this is not a shot intended at him, rather at your own logic) From jcea at jcea.es Mon Apr 26 19:57:09 2010 From: jcea at jcea.es (Jesus Cea) Date: Mon, 26 Apr 2010 19:57:09 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? Message-ID: <4BD5D3F5.7090503@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We don't have any buildbot backing this system. OSF/1 last version was in 1994, was picked by Digital (Tru64 Unix). Last version of Tru64 was released in late 2006. Now Digital is owned by HP with its own Unix (HP-UX). Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have any buildbot running any of this systems... Deprecated systems are documented in PEP-11. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS9XT9Zlgi5GaxT1NAQJkVgP9G6mj+Cm3BoXrAYQQXqfGevGWQhwQENgI ONH2B4XzOQZv9JTorDcWjU9Onf7s3YJ0AZk5pBxuekDOoblFKEz0yU3X40HLhU3N 8dxJ6aRNYBpPLYFbJp2tuGb2rs/VpfuKPAlmKxcRKUvC7heyxokFyXJnnXy2i50r XnHYwfz5PG8= =SseK -----END PGP SIGNATURE----- From steve at holdenweb.com Mon Apr 26 20:15:01 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 26 Apr 2010 14:15:01 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> Message-ID: Antoine Pitrou wrote: > Steven D'Aprano pearwood.info> writes: >> Who are we worried about offending? The crowds on the Internet who never >> volunteer for anything, who never submit patches, let alone offer to do >> the unglamourous work? > > Perhaps you should look more carefully. We do have contributors who submit > patches and advice on the tracker. There isn't just the committers and the > passive masses. > Yes, in the last year in particular there has been some excellent effort of maintaining the issue tracker content. But the question still remains - who are we worried about offending? > (oh, and following your logic, we should ignore your advice, unless you actually > contribute to the "unglamourous work" - do you?) > >> In a meritocracy it isn't enough to be >> good at what you do, you also have to be known to be good. > > If this were the criterion then the answer would be simple: nobody seems to > knows dangerjim in the Python community. > Except, of course, the person recommending him. And it seems from the discussion that nobody is particularly bothered about finding out about him, preferring to exercise their various prejudices in preference to taking a PSF member's word that he's a potentially valuable contributor along with an offer of supervision. I didn't realize we had so much effort available that we can ignore such offers. > (to make it clear: this is not a shot intended at him, rather at your own logic) > > To make it clear: this is not intended as a criticism of you personally, rather of those who do not seem to feel that increasing the developer community is important. Perhaps diversity is just something you write in a statement. Some of the comments in this thread have seemed positively unwelcoming, even though I doubt that was the authors' intention. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From tjreedy at udel.edu Mon Apr 26 20:23:00 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 14:23:00 -0400 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: <20100426061223.GA17467@tummy.com> References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: On 4/26/2010 2:12 AM, Sean Reifschneider wrote: > On Sun, Apr 25, 2010 at 08:42:00PM -0400, Terry Reedy wrote: >> What is *his* interest? How long has he known and used Python? > > Good points have been made on both sides of the issue here. Despite my > having a vested interest, I really have no strong feelings one way or > another on the initial request. Of course, much of the discussion has little to do with the particulars of your request ;-). > But, the answers to your questions above may make it more clear why I was > looking for the enhanced privileges. > > James (dangerjim) has *NO* knowledge of Python -- he has done primarily C > and Java coding. He *DOES* have time on his hands. This is why I proposed > to him to do tracker triage. > > However, as I started walking him through some examples of triage, I > realized that regular accounts don't have the privileges to do any of the > things I was proposing. For example: > > Go into the list of task tasks and look at the ones with no priority. No priority effectively means normal priority. I will separately suggest that the latter be made the default so there is no need for anyone to make such busywork changes. > Read through the description and any follow-ups and try to figure out > what priority to give it. In most cases it will be "normal". As said above, the need to do this should be fixed. In the meantime, if people really care about having 'no selection' replaced by 'normal', I could do more. I have not bothered because I regard the two as synonyms and have not bothered. What a boring thing to give to a newcomer. > However, > for some issues it will be clear they should be a higher or lower > priority, even to someone who doesn't know Python. After years on the tracker, *I* do not try to make such judgemnts, so I am dubious about the ability of a non-Python newcomer to do so terribly sensibly. > Then we went on to issue 5575 and read through it. In reading this one > to determine the priority, it was clear that the ball was back in > Collin's court, so I showed that I would look to see if Collin was a > valid assignee (which he was) and assign it to him, with a comment about > why. To my understanding, the 'asignee' is the person who will make a decision on the issue, which usually is the maintainer of the component. Who maintains the sqlite, hashlib and ssl modules? I do not know that 'asignee' should change every time the ball moves to another's court. I thought it stayed constant except when the assignee cannot deal with the issue. Is my understanding obsolete? > Go into old bugs that have patches, and see if the patches cleanly apply > against the trunk. If they do, do a "make" and "make test". Add a > comment with the outcome. This, and related C patch review activities, which do not require escalated privileges, would be *much* more useful, and probably more interesting to your C coding friend. > Two of the 3 easiest things I came up with for an outsider to help out > with, are things that his account couldn't do. But, as I said, one of those two should be fixed, and I believe auto-assignment is gradually being improved. The most useful things a C coder can do he can do now. Issues are stalled by lack of review, not by blank priority fields. Terry Jan Reedy From regebro at gmail.com Mon Apr 26 20:24:03 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Apr 2010 20:24:03 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: On Mon, Apr 26, 2010 at 17:42, R. David Murray wrote: > The first is that open source projects tend to be meritocracies. > An otherwise unknown person being introduced to the community and > immediately given privileges *just* because of the recommendation of > another person Since the recommendation is based on the persons merit, I fail to see the difference. > may feel (especially to the non privileged) like a kind > of nepotism. ?("It's not what you contribute, it's who you know"). That is only a problem if we break the rules for certain people. But this discussion isn't about breaking the rules, but changing them. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Mon Apr 26 20:27:01 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 18:27:01 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> Message-ID: Jesse Noller gmail.com> writes: > > Just to add fuel to the fire w.r.t this discussion about > process-improvements, lowering friction, etc. I'd like to point out > (unintentionally tooting my own horn) a discussion I started up on > this exact topic last week: > > http://jessenoller.com/2010/04/22/why-arent-you-contributing-to-python/ > http://www.reddit.com/r/Python/comments/burio/why_arent_you_contributing_to_python/ > http://news.ycombinator.com/item?id=1285897 I have read most of the comments there and I haven't seen anyone suggest that privileges should be given earlier or more easily. (on the other hand, a clearer process perhaps wouldn't hurt) Does your private feedback suggest otherwise? Regards Antoine. From solipsis at pitrou.net Mon Apr 26 20:30:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 18:30:55 +0000 (UTC) Subject: [Python-Dev] merit References: <4BD4A69A.4080507@tummy.com> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: Lennart Regebro gmail.com> writes: > > On Mon, Apr 26, 2010 at 17:42, R. David Murray bitdance.com> wrote: > > The first is that open source projects tend to be meritocracies. > > An otherwise unknown person being introduced to the community and > > immediately given privileges *just* because of the recommendation of > > another person > > Since the recommendation is based on the persons merit, I fail to see > the difference. In an open source community, "merit" relates to that community. We don't give Linus Torvalds all rights on the project just because we know (or assume ;-)) he is tremendously competent. Regards Antoine. From jnoller at gmail.com Mon Apr 26 20:39:11 2010 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 26 Apr 2010 14:39:11 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> Message-ID: On Mon, Apr 26, 2010 at 2:27 PM, Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> >> Just to add fuel to the fire w.r.t this discussion about >> process-improvements, lowering friction, etc. I'd like to point out >> (unintentionally tooting my own horn) a discussion I started up on >> this exact topic last week: >> >> http://jessenoller.com/2010/04/22/why-arent-you-contributing-to-python/ >> > http://www.reddit.com/r/Python/comments/burio/why_arent_you_contributing_to_python/ >> http://news.ycombinator.com/item?id=1285897 > > I have read most of the comments there and I haven't seen anyone suggest that > privileges should be given earlier or more easily. > (on the other hand, a clearer process perhaps wouldn't hurt) > > Does your private feedback suggest otherwise? I brought this up in the context of lowering the barrier to contribution; not as a value statement surrounding whether or not we should give out privileges sooner rather than later - most of the comments echo the sentiment of "it's hard or unapproachable to become part of this project". I personally feel that if a person is willing to do the work, and someone vouches for them and is willing to be their mentor, there's no reason to wave our hands and say no. jesse From tjreedy at udel.edu Mon Apr 26 20:42:15 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 14:42:15 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level Message-ID: Currently, the 'default' Priority for new tracker issues is '- no selection -'. This is, I believe, widely understood to be equivalent to 'normal'. Consequently, almost no one bothers to make a selection. This applies even to experienced people like (in the last hour) Jesus Crea (#8536), Eric Smith (*8538), and Mark Dickenson (*8540). If possible, I think 'normal' should be the default in the hox or else there should be some sort of auto replacement. Currently, there are tracker maintainers spending time doing the replacement by hand. Sean R. has even requested privileges for someone in part just so he can do so too. Such activily strikes me as silly. I think it should be made unnecessary either by improving the tracker or by being declared to be unneeded. The only time a person should set the field to change it from normal (or to reverse such a change). Terry Jan Reedy From regebro at gmail.com Mon Apr 26 20:48:35 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Apr 2010 20:48:35 +0200 Subject: [Python-Dev] merit In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: On Mon, Apr 26, 2010 at 20:30, Antoine Pitrou wrote: > Lennart Regebro gmail.com> writes: >> >> On Mon, Apr 26, 2010 at 17:42, R. David Murray bitdance.com> > wrote: >> > The first is that open source projects tend to be meritocracies. >> > An otherwise unknown person being introduced to the community and >> > immediately given privileges *just* because of the recommendation of >> > another person >> >> Since the recommendation is based on the persons merit, I fail to see >> the difference. > > In an open source community, "merit" relates to that community. We don't give > Linus Torvalds all rights on the project just because we know (or assume ;-)) he > is tremendously competent. Well, that's a blow against the merit-based position then. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From eric at trueblade.com Mon Apr 26 20:51:02 2010 From: eric at trueblade.com (Eric Smith) Date: Mon, 26 Apr 2010 14:51:02 -0400 (EDT) Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: References: Message-ID: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> > Currently, the 'default' Priority for new tracker issues is '- no > selection -'. This is, I believe, widely understood to be equivalent to > 'normal'. Consequently, almost no one bothers to make a selection. This > applies even to experienced people like (in the last hour) Jesus Crea > (#8536), Eric Smith (*8538), and Mark Dickenson (*8540). You neglected my 2 dupes at 8537 and 8539! :) > If possible, I think 'normal' should be the default in the hox or else > there should be some sort of auto replacement. Makes sense to me. Eric. From solipsis at pitrou.net Mon Apr 26 21:01:56 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 19:01:56 +0000 (UTC) Subject: [Python-Dev] Make 'normal' the default tracker priority level References: Message-ID: Terry Reedy udel.edu> writes: > > If possible, I think 'normal' should be the default in the hox or else > there should be some sort of auto replacement. Makes sense to me too. From piotr at debian.org Mon Apr 26 21:19:59 2010 From: piotr at debian.org (Piotr =?utf-8?Q?O=C5=BCarowski?=) Date: Mon, 26 Apr 2010 21:19:59 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian Message-ID: <20100426191959.GA3694@piotro.eu> Hi, Many Python module developers do not want their work to be distributed by Debian (and probably by other Linux distributions), here's a list of hints that will help you accomplish that goal: * depend on unstable or unreleased software (even if you use it only to generate docs or do unit tests), * bundle local copies of 3rd party modules and do not send your changes upstream, never document your changes in upstream code, * break API in every release, * break ABI in every second release, * do not ship all files needed to build the extension/docs/etc. in the source tarball, * do not list required dependencies in install_requires or INSTALL/README files, * if you list them, make sure you set the minimum required version to the one released yesterday, even if module works fine with version released 2 years ago, * create your own versioning schema (do not follow PEP-0386!), change it from time to time, * hardcode paths in the code and do it in as many places as you can (add new ones every few releases, that will teach them to not patch your code), * ignore FHS (you're using Windows after all); use __file__ whenever you can, * make sure there's nothing about license in LICENSE file or in file headers, * if you have some free time, create your own license, avoid BSD/MIT/(L)GPL, * if you use GPL, do not include full content of the license in the tarball, * release different tarballs with the same version number, * use waf as build-system, * release a new version without testing it with your own unit tests first to ensure that it will fail when your favourite Debian Developer tries to build it ;-) [debian-python at l.d.o BCCed] -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From steve at pearwood.info Mon Apr 26 21:22:37 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Apr 2010 05:22:37 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270330.22604.steve@pearwood.info> Message-ID: <201004270522.38521.steve@pearwood.info> On Tue, 27 Apr 2010 03:45:39 am Antoine Pitrou wrote: > Steven D'Aprano pearwood.info> writes: > > Who are we worried about offending? The crowds on the Internet who > > never volunteer for anything, who never submit patches, let alone > > offer to do the unglamourous work? > > Perhaps you should look more carefully. We do have contributors who > submit patches and advice on the tracker. There isn't just the > committers and the passive masses. Yes, I know, I'm one of those contributors who have submitted patches. Only a couple, so far, but provided I'm not discouraged by the lack of anyone with the time or motivation to review them, there will probably be more in time. (By the way, if anyone wants to review issues 4037 and 8128, I'd be grateful.) > (oh, and following your logic, we should ignore your advice, unless > you actually contribute to the "unglamourous work" - do you?) That depends on what you call unglamourous work. No, I don't triage bugs. I don't have commit privileges, so I can't. Does hand-holding newbies who don't know the difference between a list and a dict count as unglamourous work? I'm not looking for a medal, I'm just trying to give back whatever little I'm able. > > In a meritocracy it isn't enough to be > > good at what you do, you also have to be known to be good. > > If this were the criterion then the answer would be simple: nobody > seems to knows dangerjim in the Python community. Sean is nobody? We trust Sean to check in code. We trust him not to hand his username and password to dangerjim and let him loose. But do we trust his judgement that dangerjim is ready, willing and able to triage bugs? I think we can all take it as a given that commit privileges shouldn't be just given out to anyone. I think we can all agree that one good way to gain that trust is to submit lots of high-quality patches. But what I don't understand is why there is so much resistance to the idea of accepting a personal recommendation from somebody who is trusted with commit privileges, even in principle. The reasons given don't strike me as convincing, especially the idea that it is nepotistic. It's not like commit privileges is a reward that is valuable in and of itself, or that they can't be revoked if dangerjim turns out not to have the chops that Sean said. Dangerjim doesn't know Python, he can't contribute by writing patches, but he could make a valuable contribution by reviewing them. Sean has said he'll mentor him. Meritocracies are reputation-based, and the thing about reputation-based systems is that reputation propagates: I trust Alice because I've seen direct evidence of her merit, and I trust Bob on the basis that Alice vouches for him and I trust her to be a good judge of merit. Such propagation is lossy, of course. I trust Alice more than Bob. The further away from direct evidence of merit, the less confidence I have. -- Steven D'Aprano From victor.stinner at haypocalc.com Mon Apr 26 21:35:33 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 26 Apr 2010 21:35:33 +0200 Subject: [Python-Dev] Drop OS/2 support? In-Reply-To: <4BC928C8.7060007@bullseye.andymac.org> References: <201004170212.50400.victor.stinner@haypocalc.com> <4BC928C8.7060007@bullseye.andymac.org> Message-ID: <201004262135.33184.victor.stinner@haypocalc.com> Le samedi 17 avril 2010 05:19:36, Andrew MacIntyre a ?crit : > > My patch: http://bugs.python.org/issue8391 (os.execvpe() doesn't support > > surrogates in env). > > I'll look at it when I get a chance, but I'm not expecting that my input > should affect your patch and I'm not expecting you to try and deal with > the issue on OS/2. I realized that the code was broken on OS/2 because it calls bytes2str(path) whereas this function takes two mandatory arguments. My patch factorize the creation of envlist, so fixing OS/2 should now be easier. The issue is now closed. > The 3.x branch needs quite a bit of work on OS/2 to > deal with Unicode, as OS/2 was one of the earlier OSes with full > multiple language support and IBM developed a unique API. I'm still > struggling to come to terms with this, partly because I myself don't > "need" it. Ah ok, Python3 doesn't work on OS/2 :-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Mon Apr 26 21:37:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Apr 2010 19:37:31 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. References: <4BD4A69A.4080507@tummy.com> <201004270330.22604.steve@pearwood.info> <201004270522.38521.steve@pearwood.info> Message-ID: Steven D'Aprano pearwood.info> writes: > > That depends on what you call unglamourous work. No, I don't triage > bugs. I don't have commit privileges, so I can't. Is this the sole reason? You could try to review patches which are proposed, or give advice where you are competent. As Terry (I believe) said, it's certainly as useful as changing form values and checkboxes :) By the way, it isn't about commit privileges, but tracker rights. I don't think we would consider giving commit rights to someone who hasn't ever participated in the community, and apparently isn't a Python user. Regards Antoine. From ziade.tarek at gmail.com Mon Apr 26 21:39:39 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 26 Apr 2010 21:39:39 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426191959.GA3694@piotro.eu> References: <20100426191959.GA3694@piotro.eu> Message-ID: On Mon, Apr 26, 2010 at 9:19 PM, Piotr O?arowski wrote: > Hi, > > Many Python module developers do not want their work to be distributed > by Debian (and probably by other Linux distributions), here's a list of > hints that will help you accomplish that goal: Great hints, I'll try to follow them... and they could make a good section in the Hitchhiker's guide to packaging ;) [...] > * ignore FHS (you're using Windows after all); use __file__ whenever > ?you can, You should be permissive on that one. Until we know how to describe resource files properly, __file__ is what developer use when they need their projects to be portable.. Notice that we have started to work on fixing this - http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst Tarek -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Mon Apr 26 22:00:24 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Apr 2010 22:00:24 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? In-Reply-To: <4BD5D3F5.7090503@jcea.es> References: <4BD5D3F5.7090503@jcea.es> Message-ID: <4BD5F0D8.4020007@v.loewis.de> > Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have > any buildbot running any of this systems... Dropping support is fine with me, in the long term. If PEP 11 is truly followed, we should deprecate support in one version, and drop it in the next one. This would mean we add a easy-to-remove configure error in 3.2, and remove the support in 3.3. Regards, Martin From tjreedy at udel.edu Mon Apr 26 22:06:46 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 16:06:46 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> Message-ID: On 4/26/2010 2:15 PM, Steve Holden wrote: >> If this were the criterion then the answer would be simple: nobody seems to >> knows dangerjim in the Python community. >> > Except, of course, the person recommending him. And it seems from the > discussion that nobody is particularly bothered about finding out about > him, Ahem. As I indicated in my first response to Sean, the first thing I did was to check the tracker for issues listing 'damgerjim' on the nosy list. (Simce people who post messages get auto-added to such and I do not see a message versus issue search page.) I found none and asked Sean for more info. When Sean provided more, I made more particular-to-the-situation responses. I separately posted, in a new thread, a suggestion that we change the tracker to eliminate busywork rather than recruit new people to do it for us. What a way to drive new recruits away. =============== Several people have asked "what danger?". The most delicate part of triage is closing issues that should never have been opened, either because they ask questions that belong on python-list or make foolish bug claims. These come from newbies who could be driven away from Python by less than polite responses from someone with 'official' status. So, again, I think someone should actively exhibit the ability to politely respond to strangers before being given 'official' power to close issues. Terry Jan Reedy From piotr at debian.org Mon Apr 26 22:10:21 2010 From: piotr at debian.org (Piotr =?utf-8?Q?O=C5=BCarowski?=) Date: Mon, 26 Apr 2010 22:10:21 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> Message-ID: <20100426201021.GA8902@piotro.eu> [Tarek Ziad?, 2010-04-26] > Great hints, I'll try to follow them... and they could make a good > section in the Hitchhiker's guide to packaging ;) How about these two: http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html http://wiki.debian.org/GettingPackaged > [...] > > * ignore FHS (you're using Windows after all); use __file__ whenever > > ?you can, > > You should be permissive on that one. Until we know how to describe > resource files properly, > __file__ is what developer use when they need their projects to be portable.. > > Notice that we have started to work on fixing this - > http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst if there's no other way (--install-data is ignored right now, and I know you're doing a great work to change that, thanks BTW), one could always use it in *one* place and later import the result in other parts of the code (instead of using __file__ again) -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From brett at python.org Mon Apr 26 22:17:32 2010 From: brett at python.org (Brett Cannon) Date: Mon, 26 Apr 2010 13:17:32 -0700 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: On Mon, Apr 26, 2010 at 01:19, Paul Moore wrote: > On 26 April 2010 07:12, Sean Reifschneider wrote: > > Two of the 3 easiest things I came up with for an outsider to help out > > with, are things that his account couldn't do. > > Would in not therefore be reasonable to question whether the *default* > privileges be changed to allow "outsiders" to do these things, rather > than asking for escalated privileges for one individual? (Or more > likely, as well as doing so, as changing the default is likely to be a > longer discussion...) > No, the permission levels are purposefully where they are. I have personally had back-and-forths with bug submitters over the priority of an issue because they just happened to think it was more important than it really was. It gets really annoying when you start to get constant bug updates because someone chose to change the priority on their own. Besides the fields under discussion are for developers, not the bug submitters. Sean's friend can start off easily by simply commenting on an issue saying "this issue should probably have a normal priority" and someone with Developer privileges can then go in and either set it or disagree. As he does that more and more he will get noticed and recommended for elevated privileges or he can ask for them. -Brett > > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Apr 26 22:24:56 2010 From: brett at python.org (Brett Cannon) Date: Mon, 26 Apr 2010 13:24:56 -0700 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 12:01, Antoine Pitrou wrote: > Terry Reedy udel.edu> writes: > > > > If possible, I think 'normal' should be the default in the hox or else > > there should be some sort of auto replacement. > > Makes sense to me too. > Same here. Unfortunately I think that requires a code change as I can't see how to do it through the web API. -Brett > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon Apr 26 22:30:29 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 26 Apr 2010 22:30:29 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426201021.GA8902@piotro.eu> References: <20100426191959.GA3694@piotro.eu> <20100426201021.GA8902@piotro.eu> Message-ID: On Mon, Apr 26, 2010 at 10:10 PM, Piotr O?arowski wrote: > [Tarek Ziad?, 2010-04-26] >> Great hints, I'll try to follow them... and they could make a good >> section in the Hitchhiker's guide to packaging ;) > > How about these two: > http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html > http://wiki.debian.org/GettingPackaged Will check these, thx > >> [...] >> > * ignore FHS (you're using Windows after all); use __file__ whenever >> > ?you can, >> >> You should be permissive on that one. Until we know how to describe >> resource files properly, >> __file__ is what developer use when they need their projects to be portable.. >> >> Notice that we have started to work on fixing this - >> http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst > > if there's no other way (--install-data is ignored right now, and I know > you're doing a great work to change that, thanks BTW), one could always > use it in *one* place and later import the result in other parts of > the code (instead of using __file__ again) Sounds like a good advice to help packagers changing it, we'll add something about that in the guide. Also if you have some time to read it, your feedback as a packager would be appreciated http://guide.python-distribute.org From martin at v.loewis.de Mon Apr 26 22:39:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Apr 2010 22:39:39 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> Message-ID: <4BD5FA0B.8040504@v.loewis.de> >> If possible, I think 'normal' should be the default in the hox or else >> there should be some sort of auto replacement. > > Makes sense to me. I have now changed to make 'normal' the default priority for new issues. Shall I also set the priority on all past issues to normal which have them currently unset? I would do that "behind" roundup, so that it appears as if the issue was already created with that priority. That way, those issues don't appear as if they had recent activity. Regards, Martin From martin at v.loewis.de Mon Apr 26 22:41:56 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 26 Apr 2010 22:41:56 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: References: Message-ID: <4BD5FA94.4020408@v.loewis.de> > Same here. Unfortunately I think that requires a code change as I can't > see how to do it through the web API. You have to write an auditor, which I just did. Regards, Martin From eric at trueblade.com Mon Apr 26 22:41:15 2010 From: eric at trueblade.com (Eric Smith) Date: Mon, 26 Apr 2010 16:41:15 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FA0B.8040504@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> Message-ID: <4BD5FA6B.70200@trueblade.com> Martin v. L?wis wrote: >>> If possible, I think 'normal' should be the default in the hox or else >>> there should be some sort of auto replacement. >> Makes sense to me. > > I have now changed to make 'normal' the default priority for new issues. > Shall I also set the priority on all past issues to normal which have > them currently unset? I think the only risk is for issues that had a priority and were then deliberately changed to unset. Given how unlikely that is, I think you can go ahead and do this. Eric. From tjreedy at udel.edu Mon Apr 26 22:48:38 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 16:48:38 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <201004270522.38521.steve@pearwood.info> References: <4BD4A69A.4080507@tummy.com> <201004270330.22604.steve@pearwood.info> <201004270522.38521.steve@pearwood.info> Message-ID: On 4/26/2010 3:22 PM, Steven D'Aprano wrote: > That depends on what you call unglamourous work. No, I don't triage > bugs. I don't have commit privileges, so I can't. Tracker 'privileges' (responsibilities, really) are different from commit privileses. > Does hand-holding > newbies who don't know the difference between a list and a dict count > as unglamourous work? Whether on python-list or tracker issues (which possibly should not have been opened), yes. > I'm not looking for a medal, I'm just trying to > give back whatever little I'm able. I would not want you to drop hand-holding to do tracker admin. > I don't understand is why there is so much resistance to the idea of > accepting a personal recommendation from somebody who is trusted with > commit privileges, even in principle. Python tracker admins represent the core community to the larger Python world. There are two skills needed for such a responsibility. One is the ability to categorize in accord with vague norms. The other is social sensitivity in responding to strangers, especially tracker newbies. These are quite orthogonal to the ability to code or be someone's good buddy. Since one can do hard-to-repair damage, I think minimal evidence of these two needed skills is appropriate. > Dangerjim doesn't know Python, he can't contribute by writing patches, > but he could make a valuable contribution by reviewing them. Which he can do now, without tracker admin privileges. If successful, that would be more helpful than busywork admin. Actually, I can imagine a C coder writing certain C patches, given a decent spec, without really knowing Python. Terry Jan Reedy From brett at python.org Mon Apr 26 22:49:31 2010 From: brett at python.org (Brett Cannon) Date: Mon, 26 Apr 2010 13:49:31 -0700 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FA0B.8040504@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> Message-ID: On Mon, Apr 26, 2010 at 13:39, "Martin v. L?wis" wrote: > >> If possible, I think 'normal' should be the default in the hox or else > >> there should be some sort of auto replacement. > > > > Makes sense to me. > > I have now changed to make 'normal' the default priority for new issues. > Shall I also set the priority on all past issues to normal which have > them currently unset? > > I would do that "behind" roundup, so that it appears as if the issue was > already created with that priority. That way, those issues don't appear > as if they had recent activity. > I say yes. Might as well just clean up that field now while we are thinking about it. -Brett > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Apr 26 22:58:43 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 16:58:43 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FA6B.70200@trueblade.com> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <4BD5FA6B.70200@trueblade.com> Message-ID: On 4/26/2010 4:41 PM, Eric Smith wrote: > Martin v. L?wis wrote: >>>> If possible, I think 'normal' should be the default in the hox or else >>>> there should be some sort of auto replacement. >>> Makes sense to me. >> >> I have now changed to make 'normal' the default priority for new issues. Just now, I still see '- no selection -' for a new new-issue, but thank you for attending to this. This will fix a minor but long-standing annoyance. >> Shall I also set the priority on all past issues to normal which have >> them currently unset? > > I think the only risk is for issues that had a priority and were then > deliberately changed to unset. Given how unlikely that is, I think you > can go ahead and do this. To me, the priority should never be unset, so please go ahead. Terry Jan Reedy From martin at v.loewis.de Mon Apr 26 23:02:53 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Apr 2010 23:02:53 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FA6B.70200@trueblade.com> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <4BD5FA6B.70200@trueblade.com> Message-ID: <4BD5FF7D.10903@v.loewis.de> Eric Smith wrote: > Martin v. L?wis wrote: >>>> If possible, I think 'normal' should be the default in the hox or else >>>> there should be some sort of auto replacement. >>> Makes sense to me. >> >> I have now changed to make 'normal' the default priority for new issues. >> Shall I also set the priority on all past issues to normal which have >> them currently unset? > > I think the only risk is for issues that had a priority and were then > deliberately changed to unset. Given how unlikely that is, I think you > can go ahead and do this. I can also filter them out: roundup_tracker=> select distinct id from _issue, issue__journal where id=nodeid and _priority is null and action='set' and params like '%priority%'; id ------ 1404 1601 3947 5249 5273 5334 6150 6453 7540 I will try to expand this to a proper update query, but only tomorrow. Regards, Martin P.S. Interestingly, it was tjreedy who unset the priority on issue 6453 where it was previously normal :-) From tjreedy at udel.edu Mon Apr 26 23:39:37 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Apr 2010 17:39:37 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FF7D.10903@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <4BD5FA6B.70200@trueblade.com> <4BD5FF7D.10903@v.loewis.de> Message-ID: On 4/26/2010 5:02 PM, "Martin v. L?wis" wrote: > Eric Smith wrote: >> Martin v. L?wis wrote: >>>>> If possible, I think 'normal' should be the default in the hox or else >>>>> there should be some sort of auto replacement. >>>> Makes sense to me. >>> >>> I have now changed to make 'normal' the default priority for new issues. >>> Shall I also set the priority on all past issues to normal which have >>> them currently unset? >> >> I think the only risk is for issues that had a priority and were then >> deliberately changed to unset. Given how unlikely that is, I think you >> can go ahead and do this. > > I can also filter them out: > > roundup_tracker=> select distinct id from _issue, issue__journal where > id=nodeid and _priority is null and action='set' and params like > '%priority%'; > id > ------ > 1404 > 1601 > 3947 > 5249 > 5273 > 5334 > 6150 > 6453 > 7540 > > I will try to expand this to a proper update query, but only tomorrow. > > Regards, > Martin > > P.S. Interestingly, it was tjreedy who unset the priority on issue 6453 > where it was previously normal :-) An accident. As I wrote in the next message: "Must have been random glitch. I certainly did not change state and priority." David Murray replies "I wonder if your browser is doing something weird with the form field settings. Or it may have as you say been a random thing...I know that has happened to me once or twice where I refreshed the page in an odd order and reset some fields I wasn't intending to reset." 7540, 5334, 5273, 5249, 3947, 1404 are release blocker or other higher-than-normal priority issue whose high priorities were cleared when and as the issue was closed. Hirokazu Yamamoto (ocean-city), in particular, seems to think that higher-than-normal priorities should be cleared when issues are closed. I presume this is a mis-understanding. A critical open issue should just become a closed critical issue. If so, the clearings should be undone. 6150 had release-blocker set and cleared the day before closing. Not sure on this one whether it is case of above or below. It hardly matters though. 1601 had priority set to high and reverted on same day by different people. It should have been set back to normal. I am willing to fix all of the above by hand, if you want, but refrained for the moment because of the spurious message-to-nosy-list spew. Is there anyway to make unsetting not possible? Terry Jan Reedy From barry at python.org Mon Apr 26 23:45:47 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Apr 2010 17:45:47 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426191959.GA3694@piotro.eu> References: <20100426191959.GA3694@piotro.eu> Message-ID: <20100426174547.03ee0c86@heresy> On Apr 26, 2010, at 09:19 PM, Piotr O?arowski wrote: >Many Python module developers do not want their work to be distributed >by Debian (and probably by other Linux distributions), here's a list of >hints that will help you accomplish that goal: Okay, it took me a few bullet items to realize you were being sarcastic. :) I almost starting writing an email asking why anyone would seriously want that. This is a great list. Something I have been thinking about lately dovetails with this, but from a different direction. Let's ask why the problems you enumerate occur, and what can we do about them? For example, there's a nice tool called 'Quickly' that builds application templates using best practices. It is opinionated, but designed for the opportunistic programmer. I've been thinking about writing a Python application and/or library template for this which would make it easy to start a new project automating as much as possible, and doing things The Right Way. It may also be worthwhile putting very obvious carrots (rather than sticks) out there to encourage people to write packages using best practices. For example, once snakebite is operational, I could imagine some kind of automated testing suite for PyPI packages. IOW, if you play by rules A, B and C, we'll automatically grab your new package, build it, run the test suite, and put a gold star on your PyPI page. Some things of course, we can't automate, but I think the HGtP should provide clear examples of best practices, and tools to adhere to them where possible. It is okay (in fact encouraged) to be opinionated and not provide a lot of variation. Advanced developers can figure out how to do complex things correctly, but let's make it easy for the simpler 80% of packages out there. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Apr 26 23:46:55 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Apr 2010 17:46:55 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> Message-ID: <20100426174655.70685325@heresy> On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: >You should be permissive on that one. Until we know how to describe resource >files properly, __file__ is what developer use when they need their projects >to be portable.. Until then, isn't pkg_resources the best practice for this? (I'm pretty sure we've talked about this before.) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From robert.kern at gmail.com Mon Apr 26 23:56:06 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 26 Apr 2010 16:56:06 -0500 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426174655.70685325@heresy> References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> Message-ID: On 4/26/10 4:46 PM, Barry Warsaw wrote: > On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: > >> You should be permissive on that one. Until we know how to describe resource >> files properly, __file__ is what developer use when they need their projects >> to be portable.. > > Until then, isn't pkg_resources the best practice for this? (I'm pretty sure > we've talked about this before.) I don't think the OP is really speaking against using __file__ per se, but rather putting data into the package however it is accessed. The Linux-packager preferred practice is to install into the appropriate /usr/shared/ subdirectory. Writing portable libraries (with portable setup.py files!) is difficult to do that way, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ziade.tarek at gmail.com Tue Apr 27 00:03:50 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Apr 2010 00:03:50 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426174547.03ee0c86@heresy> References: <20100426191959.GA3694@piotro.eu> <20100426174547.03ee0c86@heresy> Message-ID: On Mon, Apr 26, 2010 at 11:45 PM, Barry Warsaw wrote: [..] > > For example, there's a nice tool called 'Quickly' that builds application > templates using best practices. ?It is opinionated, but designed for the > opportunistic programmer. ?I've been thinking about writing a Python > application and/or library template for this which would make it easy to start > a new project automating as much as possible, and doing things The Right Way. Related things : - Sean started "mkpkg" in distutils2, which asks you questions and creates a setup.py file. - Srid created a Paste-based script that creates a project skeleton Tarek From ziade.tarek at gmail.com Tue Apr 27 00:09:30 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Apr 2010 00:09:30 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426174655.70685325@heresy> References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> Message-ID: On Mon, Apr 26, 2010 at 11:46 PM, Barry Warsaw wrote: > On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: > >>You should be permissive on that one. Until we know how to describe resource >>files properly, __file__ is what developer use when they need their projects >>to be portable.. > > Until then, isn't pkg_resources the best practice for this? ?(I'm pretty sure > we've talked about this before.) For setuptools-based project I guess it is yes, so you can read them in zipped eggs. > > -Barry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > > -- Tarek Ziad? | http://ziade.org From barry at python.org Tue Apr 27 00:17:05 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Apr 2010 18:17:05 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <20100426174547.03ee0c86@heresy> Message-ID: <20100426181705.2bc66d74@heresy> On Apr 27, 2010, at 12:03 AM, Tarek Ziad? wrote: >On Mon, Apr 26, 2010 at 11:45 PM, Barry Warsaw wrote: >[..] >> >> For example, there's a nice tool called 'Quickly' that builds application >> templates using best practices. ?It is opinionated, but designed for the >> opportunistic programmer. ?I've been thinking about writing a Python >> application and/or library template for this which would make it easy to start >> a new project automating as much as possible, and doing things The Right Way. > >Related things : > >- Sean started "mkpkg" in distutils2, which asks you questions and >creates a setup.py file. >- Srid created a Paste-based script that creates a project skeleton I knew about Srid's but not Sean's. Thanks, I'll take a look. I don't want to stifle innovation here, but let's see if there are commonalities we can share. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Apr 27 00:21:10 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Apr 2010 18:21:10 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> Message-ID: <20100426182110.58538d12@heresy> On Apr 26, 2010, at 04:56 PM, Robert Kern wrote: >On 4/26/10 4:46 PM, Barry Warsaw wrote: >> On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: >> >>> You should be permissive on that one. Until we know how to describe resource >>> files properly, __file__ is what developer use when they need their projects >>> to be portable.. >> >> Until then, isn't pkg_resources the best practice for this? (I'm pretty sure >> we've talked about this before.) > >I don't think the OP is really speaking against using __file__ per se, but >rather putting data into the package however it is accessed. The >Linux-packager preferred practice is to install into the appropriate >/usr/shared/ subdirectory. Writing portable libraries (with portable >setup.py files!) is difficult to do that way, though. Tarek pointed to the rest page that captured some of the thinking on this developed at Pycon. There's really two sides to it - what does the programmer write and how does that integrate with the system? I really don't think the developer should go through any contortions to make it work right for the platform. For one thing, there's no way they can do it for every platform their code might end up on. It's a lose to attempt it. Tarek's design allows for separation of concerns and indirection so the programmer can worry about the parts they care about, and the platform packagers can worry about the parts they care about. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ziade.tarek at gmail.com Tue Apr 27 00:22:30 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Apr 2010 00:22:30 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> Message-ID: On Mon, Apr 26, 2010 at 11:56 PM, Robert Kern wrote: [..] > > I don't think the OP is really speaking against using __file__ per se, but > rather putting data into the package however it is accessed. The > Linux-packager preferred practice is to install into the appropriate > /usr/shared/ subdirectory. Writing portable libraries (with portable > setup.py files!) is difficult to do that way, though. Well, the only portable way to read a resource file these days in Python is to put it in your project source tree so you know for sure you can access it the same way no matter what the platform is. Thus, using __file__. The key to fix this is to have a registered list of paths for each platform at Python level. Roughly summarized : the work we are doing will consist of defining variables that points to target paths for each platform (like /usr/shared in linux) at Python level, and let the developers define that a file is under a path defined by a variable (like "shared"). For example, a project will be able to read/write the "foo.txt" file using: pkgutil.open('shared', 'foo.txt') pkgutil will look in Python for the path corresponding to "shared" and expand it. The result will vary depending on the platform, and the os packagers will be able to change that path globally in Python or locally in the project. From dmalcolm at redhat.com Tue Apr 27 01:15:48 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 26 Apr 2010 19:15:48 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426191959.GA3694@piotro.eu> References: <20100426191959.GA3694@piotro.eu> Message-ID: <1272323748.2834.15.camel@surprise> On Mon, 2010-04-26 at 21:19 +0200, Piotr O?arowski wrote: > Hi, > > Many Python module developers do not want their work to be distributed > by Debian (and probably by other Linux distributions), here's a list of Thanks! Not just Debian: I can confirm, from bitter experience, that your list is also highly applicable to Fedora and RHEL... > hints that will help you accomplish that goal: [snip] From a.badger at gmail.com Tue Apr 27 01:24:38 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 26 Apr 2010 19:24:38 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426174655.70685325@heresy> References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> Message-ID: <20100426232438.GR4350@unaka.lan> On Mon, Apr 26, 2010 at 05:46:55PM -0400, Barry Warsaw wrote: > On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: > > >You should be permissive on that one. Until we know how to describe resource > >files properly, __file__ is what developer use when they need their projects > >to be portable.. > > Until then, isn't pkg_resources the best practice for this? (I'm pretty sure > we've talked about this before.) > I would have to say no to this. Best practice from the Linux packager POV would be something like this foo/ foo/__init__.py foo/paths.py:: # Global paths where resources are installed HELPDIR='/usr/share/foo/help' TEMPLATEDIR='/usr/share/foo/templates' CACHEDIR='/var/cache/foo' DBDIR='/var/lib/foo/db' PRIVATEMODDIR='/usr/share/foo/foolib' PLUGINDIR='/usr/lib/foo/plugins' LOCALEDIR='/usr/share/locale' foo/do_things.py:: import foo.paths import os.path # connect to the db db = foo_connect(os.path.join(foo.paths.DBDIR, 'foodb.sqlite')) Using this strategy, you, the developer, can set the default paths to whatever makes the most sense for your target but the packager can go through and patch new locations in a single file that are used throughout your program. Note that you can use pkg_resources.resource_filename() in the above scenario for static resources (but not stateful ones like database files, caches, and the like). Unfortunately, pkg_resources.resource_stream() can't be adapted to the same purpose. A better version of resource_stream() based on the work that Tarek's been doing in the backend would be possible for making a resource_stream() capable of handling settable paths as it would pull the pathname determinaiton to the level behind the resource_stream call. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From prologic at shortcircuit.net.au Tue Apr 27 01:26:12 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Tue, 27 Apr 2010 09:26:12 +1000 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <1272323748.2834.15.camel@surprise> References: <20100426191959.GA3694@piotro.eu> <1272323748.2834.15.camel@surprise> Message-ID: On Tue, Apr 27, 2010 at 9:15 AM, David Malcolm wrote: > On Mon, 2010-04-26 at 21:19 +0200, Piotr O?arowski wrote: >> Many Python module developers do not want their work to be distributed >> by Debian (and probably by other Linux distributions), here's a list of > Thanks! ? Not just Debian: I can confirm, from bitter experience, that > your list is also highly applicable to Fedora and RHEL... Honestly, it's enough to publish your python application/library/module to pypi (at least this is true for my work). Re-packing some xyz Linux distribution's package mangement system just seems useless and a waste of time (with exceptions ofc). --James From fuzzyman at voidspace.org.uk Tue Apr 27 01:52:50 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 27 Apr 2010 00:52:50 +0100 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <1272323748.2834.15.camel@surprise> Message-ID: <4BD62752.1030405@voidspace.org.uk> On 27/04/2010 00:26, James Mills wrote: > On Tue, Apr 27, 2010 at 9:15 AM, David Malcolm wrote: > >> On Mon, 2010-04-26 at 21:19 +0200, Piotr O?arowski wrote: >> >>> Many Python module developers do not want their work to be distributed >>> by Debian (and probably by other Linux distributions), here's a list of >>> >> Thanks! Not just Debian: I can confirm, from bitter experience, that >> your list is also highly applicable to Fedora and RHEL... >> > Honestly, it's enough to publish your python application/library/module to pypi > (at least this is true for my work). Re-packing some xyz Linux distribution's > package mangement system just seems useless and a waste of time (with > exceptions ofc). > The point of the discussion has been how you can structure your package so that Linux distribution maintainers can repackage it *for you* whilst still remaining portable to other platforms. Even if you never intend to repackage your project yourself following these "best practises" seems like a good idea. (Unless you *really* don't want them packaged for debian - in which case you now have a handy guide telling you how you can ensure this.) All the best, Michael > --James > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From cournape at gmail.com Tue Apr 27 02:43:49 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 27 Apr 2010 09:43:49 +0900 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426201021.GA8902@piotro.eu> References: <20100426191959.GA3694@piotro.eu> <20100426201021.GA8902@piotro.eu> Message-ID: On Tue, Apr 27, 2010 at 5:10 AM, Piotr O?arowski wrote: > if there's no other way (--install-data is ignored right now, and I know > you're doing a great work to change that, thanks BTW), one could always > use it in *one* place and later import the result in other parts of > the code (instead of using __file__ again) May I ask why this is not actually the solution to resources location ? For example, let's say we have (hypothetic version of distutils supporting autoconf paths): python setup.py install --prefix=/usr --datadir=/var/lib/foo --manpath=/somefunkypath Then the install step would generate a file __install_path.py such as: PREFIX = "/usr" DATADIR = "/var/lib/foo" MANPATH = "/somfunkypath" There remains then the problem of relocatable packages, but solving this would be easy through a conditional in this generated file: if RELOCATABLE: PREFIX = "$prefix" ... else: and define $prefix and co from __file__ if necessary. All this would be an implementation detail, so that the package developer effectively do from mypkg.file_paths import PREFIX, DATADIR, etc... This is both simple and flexible: it is not mandatory, it does not make life more complicated for python developers who don't care about platform X. FWIW, that's the scheme I intend to support in my own packaging solution, cheers, David From steve at pearwood.info Tue Apr 27 03:15:49 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Apr 2010 11:15:49 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> Message-ID: <201004271115.50329.steve@pearwood.info> On Tue, 27 Apr 2010 05:37:31 am Antoine Pitrou wrote: > Steven D'Aprano pearwood.info> writes: > > That depends on what you call unglamourous work. No, I don't triage > > bugs. I don't have commit privileges, so I can't. > > Is this the sole reason? You could try to review patches which are > proposed, or give advice where you are competent. No, of course not. There are always other reasons, the biggest is too many things to do and not enough time to do it. If I did review patches, would they be accepted on the strength on my untrusted reviews? But this isn't about my reasons for not being more active on the bug tracker. This is about how we decided whether or not to give tracker rights to a volunteer. I think that being vouched for by a trusted member of good standing who offers to act as mentor should be sufficient grounds for giving tracker rights to the volunteer. Apparently some disagree. How do we resolve this, since we're obviously not persuading each other. Put it to a vote? Ask the BDFL? > By the way, it isn't about commit privileges, but tracker rights. I > don't think we would consider giving commit rights to someone who > hasn't ever participated in the community, and apparently isn't a > Python user. Correction noted, thank you. -- Steven D'Aprano From stephen at xemacs.org Tue Apr 27 04:14:20 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Apr 2010 11:14:20 +0900 Subject: [Python-Dev] Enhanced tracker privileges for "dangerjim" to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: <87hbmxk5fn.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > As said above, the need to do this should be fixed. In the meantime, if > people really care about having 'no selection' replaced by 'normal', I > could do more. I have not bothered because I regard the two as synonyms > and have not bothered. Technically they're very close to synonymous (I find it hard to imagine that people specify "priority: normal" in searches very often), but it's not synonymous to the reporter in most cases. I've had users tell me that "unselected" looks untidy, so except for assignee, where "not assigned" is very significant information, I either provide a default (which is not hard to do), or require that the user provide values in my tracker (where I've managed to reduce those fields to two, the issue summary line and the component). > What a boring thing to give to a newcomer. [...] > Issues are stalled by lack of review, not by blank priority fields. Sure, some people would be massively turned off by such duty, but others hardly mind it at all. The newcomer can always just say no. I don't think the point was to find a person to be Priority-Setter- for-Life, but rather to familiarize dangerjim with the bug tracker, the issues, and do something at least a little bit useful. Agreed, I doubt that setting priority would increase the amount of review done, since developers will generally disagree with the reporter (and non-dev third parties) about priority anyway. But getting bugs assigned to people so that they would appear in "my bugs" might help a little bit. From meadori at gmail.com Tue Apr 27 04:50:24 2010 From: meadori at gmail.com (Meador Inge) Date: Mon, 26 Apr 2010 21:50:24 -0500 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: > In other words, I think the goal is not just to add new developers to > the community, but to continue to build a strong community of developers. FWIW, from a Python community newbie that has submitted a few patches and commented on the tracker for a few months, I agree with this statement and the way things are now. I was attracted to the Python community, in part, because the development model seemed so mature and well specified. I felt that it was clear from the current documented policies on how I could contribute -- do these things and get these privileges. That simple. Moreover, by having the different stages (do these things to get tracker privileges, do these other things to get commit privileges, etc...) it was more clear how I could set personal milestones for myself to become a contributing member of the community. I find these "stages" useful for the current community to somewhat gauge an unknown person's ability, but also for that unknown person to develop and learn about the community at a reasonable pace. Yes, I know that the issue in question involves not a _completely_ unknown person, but someone who is known by an existing member of the community. However, this is about a community choice and not just one person's choice. Not to mention the fact that most anyone could have already submitted the amount of comments needed to get enhanced tracker privileges in the amount of time that has been spent on this thread :-) -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Apr 27 04:59:26 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Apr 2010 11:59:26 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> Message-ID: <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> Steve Holden writes: > Yes, in the last year in particular there has been some excellent effort > of maintaining the issue tracker content. But the question still remains > - who are we worried about offending? In this thread, we did worry about offending Sean and dangerjim. Now that Sean has commented, I don't think anybody is worrying about offending anybody; there is an understanding that there's a process issue to be resolved. The question is how best to build the community. There are two camps, the quantity camp ("low cost of contribution means more contributors, and that's good"), and the quality camp ("more interaction within the community, especially of experienced developers with newcomers, means more effective contributors and that's good"). > I didn't realize we had so much effort available that we can ignore > such offers. Steve, calm down; nobody contributing to the thread is ignoring the offer, and the status quo terms seem to be acceptable to both Sean and dangerjim. They correctly asked for more privilege so that the proposed work can be done more efficiently. Equally correctly, there is a discussion of whether an exception to past practice should be made here, and whether it's worth changing that past practice. I find RDM's argument quite compelling, that the current policy does impose some costs on the newcomer and the mentor, but that on the other hand it does strongly encourage interactions which both build community in the sense of interpersonal relationships and improve the mentoring process for the actual work being done. He concludes that the small costs involved (including the possibility of discouraging some potential contributors) are more than compensated for by the quality of the product. Where do you disagree with that logic? > To make it clear: this is not intended as a criticism of you > personally, rather of those who do not seem to feel that increasing > the developer community is important. Perhaps diversity is just > something you write in a statement. *By definition*, a community is not diverse in the most fundamental sense. As long as Pythonicity is important to Python, there is danger as well as opportunity in more rapid influx of newcomers. From stephen at xemacs.org Tue Apr 27 05:23:37 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Apr 2010 12:23:37 +0900 Subject: [Python-Dev] merit In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> <4BD5735D.7010202@voidspace.org.uk> <1272281087.3539.22.camel@localhost> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> Message-ID: <87eii1k286.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > On Mon, Apr 26, 2010 at 20:30, Antoine Pitrou wrote: > > In an open source community, "merit" relates to that community. > > We don't give Linus Torvalds all rights on the project just > > because we know (or assume ;-)) he is tremendously competent. > > Well, that's a blow against the merit-based position then. :) Not at all. Merit is determined not by "absolute" competence, but by fitness for the range of tasks to be performed. I don't really understand how someone who is not familiar with Python internals or with the Python community can be expected to have any *immediate* merit beyond "fast learner". Until a newcomer does that learning, what's the hurry in granting privileges? Note that this discussion has brought up a number of fine points about tracker work (cf Terry Reedy's posts) that Sean may not have been aware of. Had tracker privilege been granted without discussion, he still would not know about them! *Nor would I.* The discussion has had educational benefits beyond the newcomer. This is RDM's thesis about "growing the community" in action. From stephen at xemacs.org Tue Apr 27 05:46:02 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Apr 2010 12:46:02 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87d3xlk16t.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > > Sure, but that's still *work*, and it's work for *somebody else*. > > Yes, but only when the checkin was wrong. For all other checkins, it's > *less* work. Hence, a committer needs to basically fudge up every > second checkin to cause more work than he relieves work. :) Counting checkins is not the appropriate way to measure work here, and there are externalities. In my experience (in other projects, I suspect it applies to Python, too), most patches produced by newcomers scratch very personal itches that almost nobody else cares about. Many of their bugs, however, affect a large number of users. Similarly, but much less seriously, I suspect that issue triage by newcomers will not result in very "Pythonic" decisions. I won't say that setting priority or assignee inappropriately "fucks things up", but they do increase entropy of the project. Terry Reedy disagreed with Sean's judgment about setting priority and assignee, a useful discussion that would *not* have happened with the policy you propose. It might be preferable for that discussion to have happened on the tracker-discuss list, of course, but IMHO it's good for such threads to happen somewhere that tracker workers can see it. From g.brandl at gmx.net Tue Apr 27 09:18:31 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 27 Apr 2010 09:18:31 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Am 26.04.2010 15:34, schrieb Lennart Regebro: > On Mon, Apr 26, 2010 at 12:58, Stephen J. Turnbull wrote: >> It is entirely *not* evident to me that it's too hard to get >> privileges in the Python development community (Python's development >> process works -- and it works really well by comparison to 99% of the >> processes out there). > > Well, that's true, all to often a project is controlled by a few > developers with no intent of sharing access ever. > >> Sure, but that's still *work*, and it's work for *somebody else*. > > Yes, but only when the checkin was wrong. For all other checkins, it's > *less* work. Hence, a committer needs to basically fudge up every > second checkin to cause more work than he relieves work. :) Reviewing the checkins is not work, then? Georg From regebro at gmail.com Tue Apr 27 09:34:57 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 27 Apr 2010 09:34:57 +0200 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <19412.45848.576615.634858@montanaro.dyndns.org> <4BD4C6A3.3030409@palladion.com> <87ljcakh6t.fsf@uwakimon.sk.tsukuba.ac.jp> <87k4rujx9v.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Apr 27, 2010 at 09:18, Georg Brandl wrote: > Am 26.04.2010 15:34, schrieb Lennart Regebro: >> Yes, but only when the checkin was wrong. For all other checkins, it's >> *less* work. Hence, a committer needs to basically fudge up every >> second checkin to cause more work than he relieves work. :) > > Reviewing the checkins is not work, then? Well, yes, so OK, not half. But quite a lot of the checkins need to be bad for the amount of work the established commiters need to do per bug fixed to increase when you add a newbie. Yes, maybe the workload in total increases in the beginning, but that's because the newbie is actually fixing bugs, and as Stephen mentions, it's often itches no one else (read the established committers) cares about, because they didn't encounter that particular issue. I have a problem seeing that as a bad thing. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Tue Apr 27 09:41:02 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Apr 2010 09:41:02 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: <20100426232438.GR4350@unaka.lan> References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> <20100426232438.GR4350@unaka.lan> Message-ID: On Tue, Apr 27, 2010 at 1:24 AM, Toshio Kuratomi wrote: > On Mon, Apr 26, 2010 at 05:46:55PM -0400, Barry Warsaw wrote: >> On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: >> >> >You should be permissive on that one. Until we know how to describe resource >> >files properly, __file__ is what developer use when they need their projects >> >to be portable.. >> >> Until then, isn't pkg_resources the best practice for this? ?(I'm pretty sure >> we've talked about this before.) >> > I would have to say no to this. ?Best practice from the Linux packager POV > would be something like this > > foo/ > foo/__init__.py > foo/paths.py:: > > ?# Global paths where resources are installed > ?HELPDIR='/usr/share/foo/help' > ?TEMPLATEDIR='/usr/share/foo/templates' > ?CACHEDIR='/var/cache/foo' > ?DBDIR='/var/lib/foo/db' > ?PRIVATEMODDIR='/usr/share/foo/foolib' > ?PLUGINDIR='/usr/lib/foo/plugins' > ?LOCALEDIR='/usr/share/locale' > > foo/do_things.py:: > ?import foo.paths > ?import os.path > ?# connect to the db > ?db = foo_connect(os.path.join(foo.paths.DBDIR, 'foodb.sqlite')) > > Using this strategy, you, the developer, can set the default paths to > whatever makes the most sense for your target but the packager can go > through and patch new locations in a single file that are used throughout > your program. > You are making the assumption that the developers know what are the global paths on each platform. I don't think people would do that unless we provide these paths already, and that's basically the goal of the next PEP. Maybe a step toward this goal would be to provide a document that explains the role of each path, for each platform from the *python developer POV* Until then, the only approach a developer has to make sure he can access to his resource files, is to have them alongside the code. Regards Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Tue Apr 27 09:49:01 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Apr 2010 09:49:01 +0200 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <20100426201021.GA8902@piotro.eu> Message-ID: On Tue, Apr 27, 2010 at 2:43 AM, David Cournapeau wrote: > On Tue, Apr 27, 2010 at 5:10 AM, Piotr O?arowski wrote: > >> if there's no other way (--install-data is ignored right now, and I know >> you're doing a great work to change that, thanks BTW), one could always >> use it in *one* place and later import the result in other parts of >> the code (instead of using __file__ again) > > May I ask why this is not actually the solution to resources location > ? For example, let's say we have (hypothetic version of distutils > supporting autoconf paths): > > python setup.py install --prefix=/usr --datadir=/var/lib/foo > --manpath=/somefunkypath > > Then the install step would generate a file __install_path.py such as: > > PREFIX = "/usr" > DATADIR = "/var/lib/foo" > MANPATH = "/somfunkypath" > > There remains then the problem of relocatable packages, but solving > this would be easy through a conditional in this generated file: > > if RELOCATABLE: > ? ?PREFIX = "$prefix" > ? ?... > else: > > and define $prefix and co from __file__ if necessary. All this would > be an implementation detail, so that the package developer effectively > do > > from mypkg.file_paths import PREFIX, DATADIR, etc... > > This is both simple and flexible: it is not mandatory, it does not > make life more complicated for python developers who don't care about > platform X. FWIW, that's the scheme I intend to support in my own > packaging solution, That resembles a lot to what we want to achieve in the next PEP: at installation time, a file that contains all the prefix will be generated, combined with a global list of variables found in Python. Then, instead of importing these values (in our plans, the variables are statically defined), developers will do: pkgutil.open('MANPATH', 'foo'), where foo.txt is a file under /somefunkypath in your example Since you are building your own tool, it would be great to have you working with us in the upcoming PEP, since it aims to provide an interoperability ground to install and consume resource files. Regards Tarek -- Tarek Ziad? | http://ziade.org From rdmurray at bitdance.com Tue Apr 27 14:25:37 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 08:25:37 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> Message-ID: <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> On Mon, 26 Apr 2010 14:15:01 -0400, Steve Holden wrote: > Antoine Pitrou wrote: > > Steven D'Aprano pearwood.info> writes: > >> Who are we worried about offending? The crowds on the Internet who never > >> volunteer for anything, who never submit patches, let alone offer to do > >> the unglamourous work? > > > > Perhaps you should look more carefully. We do have contributors who submit > > patches and advice on the tracker. There isn't just the committers and the > > passive masses. > > > Yes, in the last year in particular there has been some excellent effort > of maintaining the issue tracker content. But the question still remains > - who are we worried about offending? The people who are potential new contributors but don't currently know anyone in the Python community. > > (oh, and following your logic, we should ignore your advice, unless you actually > > contribute to the "unglamourous work" - do you?) > > > >> In a meritocracy it isn't enough to be > >> good at what you do, you also have to be known to be good. > > > > If this were the criterion then the answer would be simple: nobody seems to > > knows dangerjim in the Python community. > > > Except, of course, the person recommending him. And it seems from the > discussion that nobody is particularly bothered about finding out about > him, preferring to exercise their various prejudices in preference to > taking a PSF member's word that he's a potentially valuable contributor > along with an offer of supervision. > > I didn't realize we had so much effort available that we can ignore such > offers. This discussion has never been about dangerjim's qualifications, as far as I can tell. I believe we all fully expect him to be a valuable contributor within a very short time, because Sean is recommending him, and we welcome him to the community, The discussion, in my view, is about the process in general, and how to make sure that it continues to promote good, inclusive community, by holding everyone to the same standards. (And the discussion, then, is should we change the current standard.) > > (to make it clear: this is not a shot intended at him, rather at your own logic) > > > > > To make it clear: this is not intended as a criticism of you personally, > rather of those who do not seem to feel that increasing the developer > community is important. Perhaps diversity is just something you write in > a statement. > > Some of the comments in this thread have seemed positively unwelcoming, > even though I doubt that was the authors' intention. I have not read any of the comments as unwelcoming (although I could be misremembering), so I'm not sure why you heard that. We are talking about process and what works best for community building (which includes increasing the number of people in the developer community). And I at least am in the mode of *discussing* it, not speaking from a position set in stone...if the consensus that develops is that the familiarization period can be skipped in certain cases, I'm not going to block that consensus or get mad about it...but I don't think we have a developing consensus right now, and I'm not sure how to move forward on that. For the record, note that both Antoine and I have been instrumental in bringing more than one new person into both the triage and the committer ranks. We (along with others) *are* the ones doing the welcoming and the mentoring and the growing of the developer community. -- R. David Murray www.bitdance.com From solipsis at pitrou.net Tue Apr 27 14:31:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Apr 2010 12:31:52 +0000 (UTC) Subject: [Python-Dev] spam on b.p.o References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> Message-ID: Terry Reedy udel.edu> writes: > > On 4/26/2010 11:09 AM, Alexander Belopolsky wrote: > > > > I also I don't remember ever seeing spam in the bugs.python.org > > comments which suggests that the subscription process weeds bots > > reasonably well. > > And when it fails, spam is deleted just so no one does see it. Speaking of which, what is the procedure to delete a spam message and remove a spamming user? We have an example here: http://bugs.python.org/user12283 Regards Antoine. From steve at holdenweb.com Tue Apr 27 14:34:45 2010 From: steve at holdenweb.com (Steve Holden) Date: Tue, 27 Apr 2010 08:34:45 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> Message-ID: <4BD6D9E5.2020005@holdenweb.com> R. David Murray wrote: > On Mon, 26 Apr 2010 14:15:01 -0400, Steve Holden wrote: [...] > For the record, note that both Antoine and I have been instrumental in > bringing more than one new person into both the triage and the committer > ranks. We (along with others) *are* the ones doing the welcoming and > the mentoring and the growing of the developer community. > For which work I am truly grateful, as I am sure are many others. Please forgive any prickliness I may have evinced in this conversation. It *is* important to make people feel welcome, and I am happy to see the development community growing. As regards the procedural discussions, while I may have my opinions it's clearly best if the procedures are maintained by those operating them. I am usually fine with that happening, and this is no exception. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From solipsis at pitrou.net Tue Apr 27 14:41:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Apr 2010 12:41:19 +0000 (UTC) Subject: [Python-Dev] spam on b.p.o References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> Message-ID: Antoine Pitrou pitrou.net> writes: > > Speaking of which, what is the procedure to delete a spam message and remove a > spamming user? Well, for some reason I hadn't seen the "remove button" message... As for deleting the user, I suppose an admin can do it. Regards Antoine. From rdmurray at bitdance.com Tue Apr 27 14:48:40 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 08:48:40 -0400 Subject: [Python-Dev] Assigned To field usage (was: Enhanced tracker privileges) In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> Message-ID: <20100427124840.4034D1FC6D7@kimball.webabinitio.net> On Mon, 26 Apr 2010 14:23:00 -0400, Terry Reedy wrote: > On 4/26/2010 2:12 AM, Sean Reifschneider wrote: > > Then we went on to issue 5575 and read through it. In reading this one > > to determine the priority, it was clear that the ball was back in > > Collin's court, so I showed that I would look to see if Collin was a > > valid assignee (which he was) and assign it to him, with a comment about > > why. > > To my understanding, the 'asignee' is the person who will make a > decision on the issue, which usually is the maintainer of the component. > Who maintains the sqlite, hashlib and ssl modules? I do not know that > 'asignee' should change every time the ball moves to another's court. I > thought it stayed constant except when the assignee cannot deal with the > issue. > > Is my understanding obsolete? Well, in my recent experience there are two things the assignee gets used for. The first is someone claiming an issue, saying, in effect, I'm going to work this issue until it is closed. The other is to do exactly what Sean did, assign it to the next person whose decision or input is needed in order to move the issue forward. However, as you say, I think the latter is done generally when the issue *can't* move forward without that person's input (or at least not without giving them a significant opportunity to provide input). Usually this is done by the person who previously had the issue assigned to them. My perception is that making someone nosy on an issue is preferred to assigning it to them (allowing them to assign it to themselves if they think that is appropriate), unless the issue is of higher priority or someone actively working on the issue really needs the other person's input in order to move forward. But these are my own rules of thumb, and a discussion of how best to use this field may be appropriate. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Tue Apr 27 15:03:05 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 09:03:05 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD5FA0B.8040504@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> Message-ID: <20100427130305.2C0711FFF13@kimball.webabinitio.net> On Mon, 26 Apr 2010 22:39:39 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > >> If possible, I think 'normal' should be the default in the hox or else > >> there should be some sort of auto replacement. > > > > Makes sense to me. > > I have now changed to make 'normal' the default priority for new issues. > Shall I also set the priority on all past issues to normal which have > them currently unset? > > I would do that "behind" roundup, so that it appears as if the issue was > already created with that priority. That way, those issues don't appear > as if they had recent activity. +1 This will make the default grouping more useful, since now critical issues will appear at the top, instead of several pages in! -- R. David Murray www.bitdance.com From ncoghlan at gmail.com Tue Apr 27 15:34:48 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Apr 2010 23:34:48 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> Message-ID: <4BD6E7F8.5010208@gmail.com> R. David Murray wrote: > And I at > least am in the mode of *discussing* it, not speaking from a position set > in stone...if the consensus that develops is that the familiarization > period can be skipped in certain cases, I'm not going to block that > consensus or get mad about it...but I don't think we have a developing > consensus right now, and I'm not sure how to move forward on that. Having a recommendation officially mean accelerating-but-not-skipping the familiarisation period doesn't seem to have met with any significant objections. We basically do that anyway, this would just mean adding a note about it to the "getting involved" documentation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Tue Apr 27 15:38:14 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 09:38:14 -0400 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <201004271115.50329.steve@pearwood.info> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> Message-ID: <20100427133814.785331FFC99@kimball.webabinitio.net> On Tue, 27 Apr 2010 11:15:49 +1000, Steven D'Aprano wrote: > No, of course not. There are always other reasons, the biggest is too > many things to do and not enough time to do it. If I did review > patches, would they be accepted on the strength on my untrusted > reviews? It is very very helpful for *anyone* to review patches. Let's see if I can clarify the process a little. (This is, of course, my take on it, others can chime in if they think I got anything wrong.) Someone submits a bug. Someone submits a patch to fix that bug (or add the enhancement). Is that patch ready for commit? No. Is it ready for *commit review* (ie: someone with commit privileges to look at it with an eye toward committing it)? Probably not. What makes a patch ready for commit review? The patch should: 1) conform to pep 7/8 2) have unit tests that fail before the patch and succeed after 3) have documentation updates if needed 4) have a py3k port *if and only if* the port is non-trivial (well, if someone wants to add one when it is trivial that's OK, but it probably won't get used) 5) if it is at all likely to have system dependencies, be tested on at least linux and windows Making sure that these items are true does not require any in-depth expertise. In many cases it doesn't even require much time. 'Trusted' or 'untrusted' doesn't really come in to it, though doing these sorts of reviews will build trust. If you can in addition look at the patch content and critique it, so much the better. Again, *any* critique is useful, even if you can't review the whole patch in detail, because it gets it that much closer to being commit ready. And there are enough uncommitted patches in the tracker that it ought to be possible for almost anyone to find something they can usefully do a content critique on. The goal is to make the commit review step as simple and fast for the committer as possible. The more eyes on the patch before hand, the faster the commit review will be. And those people who do a good job making patches commit ready will be on the fast track to getting commit privileges. -- R. David Murray www.bitdance.com PS: note that I'm using 'commit review' above with a different sense than that value is currently defined to have in the workflow. I'm thinking about advocating that the definition in the workflow be changed, and indeed we (the informal triage crew) have already occasionally used that setting with the meaning I give it above. From ncoghlan at gmail.com Tue Apr 27 15:38:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Apr 2010 23:38:30 +1000 Subject: [Python-Dev] Assigned To field usage In-Reply-To: <20100427124840.4034D1FC6D7@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100427124840.4034D1FC6D7@kimball.webabinitio.net> Message-ID: <4BD6E8D6.6020907@gmail.com> R. David Murray wrote: > Well, in my recent experience there are two things the assignee gets > used for. The first is someone claiming an issue, saying, in effect, > I'm going to work this issue until it is closed. The other is to do > exactly what Sean did, assign it to the next person whose decision or > input is needed in order to move the issue forward. However, as you > say, I think the latter is done generally when the issue *can't* move > forward without that person's input (or at least not without giving them > a significant opportunity to provide input). Usually this is done by > the person who previously had the issue assigned to them. > > My perception is that making someone nosy on an issue is preferred to > assigning it to them (allowing them to assign it to themselves if they > think that is appropriate), unless the issue is of higher priority or > someone actively working on the issue really needs the other person's > input in order to move forward. But these are my own rules of thumb, > and a discussion of how best to use this field may be appropriate. That sounds like a fair description of the way I use it as well. The most common case where I will assign a bug directly to someone is if I want a yea or nay from the release manager in deciding whether or not something is acceptable for inclusion in a beta or rc release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Tue Apr 27 16:10:29 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 10:10:29 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <4BD6E7F8.5010208@gmail.com> References: <4BD4A69A.4080507@tummy.com> <4BD57CDE.7080705@voidspace.org.uk> <20100426154210.EF0C11FBFDE@kimball.webabinitio.net> <201004270330.22604.steve@pearwood.info> <20100427122537.E0CBA1FFC99@kimball.webabinitio.net> <4BD6E7F8.5010208@gmail.com> Message-ID: <20100427141029.B3B8A1FFC91@kimball.webabinitio.net> On Tue, 27 Apr 2010 23:34:48 +1000, Nick Coghlan wrote: > R. David Murray wrote: > > And I at > > least am in the mode of *discussing* it, not speaking from a position set > > in stone...if the consensus that develops is that the familiarization > > period can be skipped in certain cases, I'm not going to block that > > consensus or get mad about it...but I don't think we have a developing > > consensus right now, and I'm not sure how to move forward on that. > > Having a recommendation officially mean accelerating-but-not-skipping > the familiarisation period doesn't seem to have met with any significant > objections. > > We basically do that anyway, this would just mean adding a note about it > to the "getting involved" documentation. Sounds good to me. -- R. David Murray www.bitdance.com From daniel at stutzbachenterprises.com Tue Apr 27 16:22:07 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 27 Apr 2010 09:22:07 -0500 Subject: [Python-Dev] bug tracker permissions request Message-ID: May I have enhanced permissions on the bug tracker, so that I can perform the following tasks? - Assign issues to myself that I plan to write a patch for - Update the Stage to "patch review" after writing a patch - Occasional bug triage My login name on the tracker is "stutzbach". I only find the time to produce patches once in awhile, but when I have the time I usually produce more than one. Assigning bugs to myself will increase my motivation to write patches, as I will feel that I've made a commitment to fixing them. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Tue Apr 27 16:35:43 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 27 Apr 2010 10:35:43 -0400 Subject: [Python-Dev] what to do if you don't want your module in Debian In-Reply-To: References: <20100426191959.GA3694@piotro.eu> <20100426174655.70685325@heresy> <20100426232438.GR4350@unaka.lan> Message-ID: <20100427143542.GS4350@unaka.lan> On Tue, Apr 27, 2010 at 09:41:02AM +0200, Tarek Ziad? wrote: > On Tue, Apr 27, 2010 at 1:24 AM, Toshio Kuratomi wrote: > > On Mon, Apr 26, 2010 at 05:46:55PM -0400, Barry Warsaw wrote: > >> On Apr 26, 2010, at 09:39 PM, Tarek Ziad? wrote: > >> > >> >You should be permissive on that one. Until we know how to describe resource > >> >files properly, __file__ is what developer use when they need their projects > >> >to be portable.. > >> > >> Until then, isn't pkg_resources the best practice for this? ?(I'm pretty sure > >> we've talked about this before.) > >> > > I would have to say no to this. ?Best practice from the Linux packager POV > > would be something like this > > > > foo/ > > foo/__init__.py > > foo/paths.py:: > > > > ?# Global paths where resources are installed > > ?HELPDIR='/usr/share/foo/help' > > ?TEMPLATEDIR='/usr/share/foo/templates' > > ?CACHEDIR='/var/cache/foo' > > ?DBDIR='/var/lib/foo/db' > > ?PRIVATEMODDIR='/usr/share/foo/foolib' > > ?PLUGINDIR='/usr/lib/foo/plugins' > > ?LOCALEDIR='/usr/share/locale' > > > > foo/do_things.py:: > > ?import foo.paths > > ?import os.path > > ?# connect to the db > > ?db = foo_connect(os.path.join(foo.paths.DBDIR, 'foodb.sqlite')) > > > > Using this strategy, you, the developer, can set the default paths to > > whatever makes the most sense for your target but the packager can go > > through and patch new locations in a single file that are used throughout > > your program. > > > > You are making the assumption that the developers know what are the > global paths on each platform. > No, I'm not. The developer needs to establish sane categories, but doesn't need to worry about the exact paths. For instance, this would be perfectly fine: foo/path.py:: HELPDIR=os.path.join(os.dirname(__file__), 'help') TEMPLATEDIR=pkg_resources.resource_filename('foo', 'templates') CACHEDIR=os.path.join(os.environ.get('HOME', '/tmp'), 'foocache') Then the individual platform packagers can patch the single file, paths.py according to the neecds oftheir platform. > I don't think people would do that unless we > provide these paths already, and that's basically the goal of the next PEP. > s/paths/categories/ and I'll agree. As I said, the PEP does a lot of things right in this area. We're able to push decisions about filesystem paths out to a higher level using the PEP whereas the current state of the art has us needing to figure it all out as individual developers :-( [...] > > Until then, the only approach a developer has to make sure he can access to his > resource files, is to have them alongside the code. > I don't think so -- but this scheme certainly allows that. I think that many developers who are targeting Linux will find it more natural to specify FHS compliant paths for their files. Someone who is developing an app to be portable will likely find that placing files alongside code is more natural (although even this isn't truly portable -- CACHEDIR and other stateful files will break under the assumption that you can write to a file/directory alongside the module). And like I say, this is just about the best workaround available now. Implementation of the PEP makes this area much better. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From exarkun at twistedmatrix.com Tue Apr 27 16:40:19 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 27 Apr 2010 14:40:19 -0000 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427133814.785331FFC99@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> On 01:38 pm, rdmurray at bitdance.com wrote: >On Tue, 27 Apr 2010 11:15:49 +1000, Steven D'Aprano > wrote: >>No, of course not. There are always other reasons, the biggest is too >>many things to do and not enough time to do it. If I did review >>patches, would they be accepted on the strength on my untrusted >>reviews? > >It is very very helpful for *anyone* to review patches. Let's see if >I can clarify the process a little. (This is, of course, my take >on it, others can chime in if they think I got anything wrong.) > >Someone submits a bug. Someone submits a patch to fix that bug (or add >the enhancement). Is that patch ready for commit? No. Is it ready >for *commit review* (ie: someone with commit privileges to look at it >with an eye toward committing it)? Probably not. > >What makes a patch ready for commit review? The patch should: > > 1) conform to pep 7/8 > 2) have unit tests that fail before the patch and succeed after > 3) have documentation updates if needed > 4) have a py3k port *if and only if* the port is non-trivial > (well, if someone wants to add one when it is trivial that's OK, > but it probably won't get used) > 5) if it is at all likely to have system dependencies, be tested > on at least linux and windows This list would make a good addition to one of the cpython development pages. If potential contributors could find this information, then they'd be much more likely to participate by doing reviews. Jean-Paul From brian.curtin at gmail.com Tue Apr 27 16:56:09 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 27 Apr 2010 09:56:09 -0500 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: References: Message-ID: On Tue, Apr 27, 2010 at 09:22, Daniel Stutzbach < daniel at stutzbachenterprises.com> wrote: > May I have enhanced permissions on the bug tracker, so that I can perform > the following tasks? > > - Assign issues to myself that I plan to write a patch for > - Update the Stage to "patch review" after writing a patch > - Occasional bug triage > > My login name on the tracker is "stutzbach". > > I only find the time to produce patches once in awhile, but when I have the > time I usually produce more than one. Assigning bugs to myself will > increase my motivation to write patches, as I will feel that I've made a > commitment to fixing them. > -- > Daniel Stutzbach, Ph.D. > President, Stutzbach Enterprises, LLC > +1 from me. Daniel has had good input on many issues, has written quality patches, and if this would help him do more, I'm for it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Apr 27 17:14:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 11:14:47 -0400 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: References: Message-ID: <20100427151447.B5F492001AD@kimball.webabinitio.net> On Tue, 27 Apr 2010 09:22:07 -0500, Daniel Stutzbach wrote: > May I have enhanced permissions on the bug tracker, so that I can perform > the following tasks? Done. I agree with Brian, Daniel has been making valuable contributions for quite some time now. I/we will keep an eye on his triage, of course. -- R. David Murray www.bitdance.com From barry at python.org Tue Apr 27 17:16:51 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Apr 2010 11:16:51 -0400 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> Message-ID: <20100427111651.738d2381@heresy> On Apr 27, 2010, at 02:40 PM, exarkun at twistedmatrix.com wrote: >On 01:38 pm, rdmurray at bitdance.com wrote: >> 2) have unit tests that fail before the patch and succeed after > >This list would make a good addition to one of the cpython development >pages. If potential contributors could find this information, then >they'd be much more likely to participate by doing reviews. It would be kind of cool if there were some best practices for running said unittest both with and without the patch enabled. Kind of like using #ifdefs in C but without all the commenting-out-commenting-in error proneness. I guess you could do something like if os.getenv('BUG1234'): # Patch the frobnicator to not bloviate. Maybe more trouble than it's worth, and not always feasible of course, but I'm wondering how (or maybe if) people do things this way. With Bazaar, I often use a loom with two threads - a bottom one that contains the test that fails, and a top one that contains the fix for the test. It's a great way to develop a patch, but you lose that once you flatten the code for review. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From daniel at stutzbachenterprises.com Tue Apr 27 17:34:16 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 27 Apr 2010 10:34:16 -0500 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: <20100427151447.B5F492001AD@kimball.webabinitio.net> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> Message-ID: On Tue, Apr 27, 2010 at 10:14 AM, R. David Murray wrote: > Done. I agree with Brian, Daniel has been making valuable > contributions for quite some time now. I/we will keep an eye on > his triage, of course. > Thanks. Is there a document that describes the meaning of all of the different fields in the bug tracker? I've read http://www.python.org/dev/workflow/, but it doesn't cover everything. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Apr 27 17:43:13 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 11:43:13 -0400 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427111651.738d2381@heresy> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: <20100427154313.7E858200656@kimball.webabinitio.net> On Tue, 27 Apr 2010 11:16:51 -0400, Barry Warsaw wrote: > It would be kind of cool if there were some best practices for running said > unittest both with and without the patch enabled. Kind of like using #ifdefs > in C but without all the commenting-out-commenting-in error proneness. I > guess you could do something like > > if os.getenv('BUG1234'): > # Patch the frobnicator to not bloviate. > > Maybe more trouble than it's worth, and not always feasible of course, but I'm > wondering how (or maybe if) people do things this way. > > With Bazaar, I often use a loom with two threads - a bottom one that contains > the test that fails, and a top one that contains the fix for the test. It's a > great way to develop a patch, but you lose that once you flatten the code for > review. Well, the way I do it for review is brute force: I download the patch, delete everything except the unit test, apply that, run it, revert, apply the original patch, run it. For developing, I generally write the unit test first , but when the fix is confined to one file I can just revert the file for testing the tests while keeping the fixed copy in my edit buffer (or a save file if I'm feeling paranoid, like when it is a substantial fix). For more complex fixes I generate separate patch files for the tests and the fix as a whole, and do a revert-patch-revert-patch dance to test things. I wonder if it would be better to encourage people to post the unit tests and the fix as separate patch files. -- R. David Murray www.bitdance.com From ezio.melotti at gmail.com Tue Apr 27 17:57:57 2010 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 27 Apr 2010 17:57:57 +0200 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427111651.738d2381@heresy> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: On Tue, Apr 27, 2010 at 5:16 PM, Barry Warsaw wrote: > On Apr 27, 2010, at 02:40 PM, exarkun at twistedmatrix.com wrote: > > >On 01:38 pm, rdmurray at bitdance.com wrote: > > >> 2) have unit tests that fail before the patch and succeed after > > > >This list would make a good addition to one of the cpython development > >pages. If potential contributors could find this information, then > >they'd be much more likely to participate by doing reviews. > > It would be kind of cool if there were some best practices for running said > unittest both with and without the patch enabled. Kind of like using > #ifdefs > in C but without all the commenting-out-commenting-in error proneness. I > guess you could do something like > > if os.getenv('BUG1234'): > # Patch the frobnicator to not bloviate. > > When I'm writing the patch it's usually easy, I write the tests, see that they fail, write the fix, see that they pass. When I'm reviewing the patch, I apply the patch, see that the tests pass, svn revert the fix, check that they fail. Most of the patches affect just a couple of files, so applying the whole patch and then revert is usually trivial and probably easier than having to deal with two separate files for patch and tests. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Apr 27 18:16:34 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 27 Apr 2010 12:16:34 -0400 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: References: <20100427151447.B5F492001AD@kimball.webabinitio.net> Message-ID: <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> On Tue, 27 Apr 2010 10:34:16 -0500, Daniel Stutzbach wrote: > Thanks. Is there a document that describes the meaning of all of the > different fields in the bug tracker? > > I've read http://www.python.org/dev/workflow/, but it doesn't cover > everything. I think it is on Brett's list to update that doc, but maybe we should help him out :). Can you list what's missing? We should fill in the gaps. -- R. David Murray www.bitdance.com From barry at python.org Tue Apr 27 19:46:18 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Apr 2010 13:46:18 -0400 Subject: [Python-Dev] Two small PEP ideas Message-ID: <20100427134618.564e3e3b@heresy> I have two somewhat unrelated thoughts about PEPs. * Accepted: header When PEP 3147 was accepted, I had a few folks ask that this be recorded in the PEP by including a link to the BDFL pronouncement email. I realized that there's no formal way to express this in a PEP, and many PEPs in fact don't record more than the fact that it was accepted. I'd like to propose officially adding an Accepted: header which should include a URL to the email or other web resource where the PEP is accepted. I've come as close as possible to this (without modifying the supporting scripts or PEP 1) in PEP 3147: http://www.python.org/dev/peps/pep-3147/ I'd be willing to update things if there are no objections. * EOL schedule for older releases. We have semi-formal policies for the lifetimes of Python releases, though I'm not sure this policy is written down in any of the existing informational PEPs. However, we have release schedule PEPs going back to Python 1.6. It seems reasonable to me that we include end-of-life information in those PEPs. For example, we could state that Python 2.4 is no longer even being maintained for security, and we could state the projected date that Python 2.6 will go into security-only maintenance mode. I would not mandate that we go back and update all previous PEPs for either of these ideas. We'd adopt them moving forward and allow anyone who's motivated to backfill information opportunistically. Thoughts? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From daniel at stutzbachenterprises.com Tue Apr 27 19:46:53 2010 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 27 Apr 2010 12:46:53 -0500 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> Message-ID: On Tue, Apr 27, 2010 at 11:16 AM, R. David Murray wrote: > I think it is on Brett's list to update that doc, but maybe we should help > him out :). Can you list what's missing? We should fill in the gaps. > Sure, here's what I've noticed: The page doesn't document the Resolution or Status fields. For the Keywords field, the page only documents the "easy" keyword. Also, some of the headings in the page are enclosed in square brackets, while others are not. It's not clear to me what the brackets are intended to designate. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Apr 27 19:50:56 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Apr 2010 13:50:56 -0400 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427154313.7E858200656@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> <20100427154313.7E858200656@kimball.webabinitio.net> Message-ID: <20100427135056.5067bb54@heresy> On Apr 27, 2010, at 11:43 AM, R. David Murray wrote: >I wonder if it would be better to encourage people to post the unit >tests and the fix as separate patch files. I think this is not bad idea for larger fixes, where it's not trivial to manually edit the diff. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brett at python.org Tue Apr 27 20:15:43 2010 From: brett at python.org (Brett Cannon) Date: Tue, 27 Apr 2010 11:15:43 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100427134618.564e3e3b@heresy> References: <20100427134618.564e3e3b@heresy> Message-ID: Sounds good to me (from my phone on my way to WWW2010). On Apr 27, 2010 10:49 AM, "Barry Warsaw" wrote: I have two somewhat unrelated thoughts about PEPs. * Accepted: header When PEP 3147 was accepted, I had a few folks ask that this be recorded in the PEP by including a link to the BDFL pronouncement email. I realized that there's no formal way to express this in a PEP, and many PEPs in fact don't record more than the fact that it was accepted. I'd like to propose officially adding an Accepted: header which should include a URL to the email or other web resource where the PEP is accepted. I've come as close as possible to this (without modifying the supporting scripts or PEP 1) in PEP 3147: http://www.python.org/dev/peps/pep-3147/ I'd be willing to update things if there are no objections. * EOL schedule for older releases. We have semi-formal policies for the lifetimes of Python releases, though I'm not sure this policy is written down in any of the existing informational PEPs. However, we have release schedule PEPs going back to Python 1.6. It seems reasonable to me that we include end-of-life information in those PEPs. For example, we could state that Python 2.4 is no longer even being maintained for security, and we could state the projected date that Python 2.6 will go into security-only maintenance mode. I would not mandate that we go back and update all previous PEPs for either of these ideas. We'd adopt them moving forward and allow anyone who's motivated to backfill information opportunistically. Thoughts? -Barry _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Tue Apr 27 20:37:18 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 27 Apr 2010 14:37:18 -0400 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: <20100427133814.785331FFC99@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 R. David Murray wrote: > On Tue, 27 Apr 2010 11:15:49 +1000, Steven D'Aprano wrote: >> No, of course not. There are always other reasons, the biggest is too >> many things to do and not enough time to do it. If I did review >> patches, would they be accepted on the strength on my untrusted >> reviews? > > It is very very helpful for *anyone* to review patches. Let's see if > I can clarify the process a little. (This is, of course, my take > on it, others can chime in if they think I got anything wrong.) > > Someone submits a bug. Someone submits a patch to fix that bug (or add > the enhancement). Is that patch ready for commit? No. Is it ready > for *commit review* (ie: someone with commit privileges to look at it > with an eye toward committing it)? Probably not. > > What makes a patch ready for commit review? The patch should: > > 1) conform to pep 7/8 > 2) have unit tests that fail before the patch and succeed after > 3) have documentation updates if needed > 4) have a py3k port *if and only if* the port is non-trivial > (well, if someone wants to add one when it is trivial that's OK, > but it probably won't get used) > 5) if it is at all likely to have system dependencies, be tested > on at least linux and windows This is an excellent set of guidelines. The only drawback I see here is that the current VCS situation makes doing the review more tedious than it should be, especially for non-committers. Or maybe the Hg mirrors are truly up-to-date and working? Last I looked, they were lagging or unavailable. > Making sure that these items are true does not require any in-depth > expertise. In many cases it doesn't even require much time. 'Trusted' > or 'untrusted' doesn't really come in to it, though doing these sorts > of reviews will build trust. If you can in addition look at the patch > content and critique it, so much the better. Again, *any* critique is > useful, even if you can't review the whole patch in detail, because it > gets it that much closer to being commit ready. And there are enough > uncommitted patches in the tracker that it ought to be possible for almost > anyone to find something they can usefully do a content critique on. > > The goal is to make the commit review step as simple and fast for the > committer as possible. The more eyes on the patch before hand, the > faster the commit review will be. And those people who do a good job > making patches commit ready will be on the fast track to getting commit > privileges. > > -- > R. David Murray www.bitdance.com > > PS: note that I'm using 'commit review' above with a different sense than > that value is currently defined to have in the workflow. I'm thinking > about advocating that the definition in the workflow be changed, and > indeed we (the informal triage crew) have already occasionally used that > setting with the meaning I give it above. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvXLtkACgkQ+gerLs4ltQ79DACbB35/XFGyiYjd79OtTx+kgoNl mcsAnA4TNlM1ARjyrDrQIwv4KG48w/7h =1hGI -----END PGP SIGNATURE----- From tseaver at palladion.com Tue Apr 27 20:37:23 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 27 Apr 2010 14:37:23 -0400 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: <20100427111651.738d2381@heresy> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On Apr 27, 2010, at 02:40 PM, exarkun at twistedmatrix.com wrote: > >> On 01:38 pm, rdmurray at bitdance.com wrote: > >>> 2) have unit tests that fail before the patch and succeed after >> This list would make a good addition to one of the cpython development >> pages. If potential contributors could find this information, then >> they'd be much more likely to participate by doing reviews. > > It would be kind of cool if there were some best practices for running said > unittest both with and without the patch enabled. Kind of like using #ifdefs > in C but without all the commenting-out-commenting-in error proneness. I > guess you could do something like > > if os.getenv('BUG1234'): > # Patch the frobnicator to not bloviate. > > Maybe more trouble than it's worth, and not always feasible of course, but I'm > wondering how (or maybe if) people do things this way. > > With Bazaar, I often use a loom with two threads - a bottom one that contains > the test that fails, and a top one that contains the fix for the test. It's a > great way to develop a patch, but you lose that once you flatten the code for > review. You can always "shelve" the part of the patch which isn't the test: I do that pretty frequently in the Zope tree, where I am now doing most development with bzr. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvXLuMACgkQ+gerLs4ltQ5HBQCgw7kqJ52kPz+0cwNSpyVUkCFA yQUAoLHJiYi+59Cc7BCeL46hA+Wygo66 =93vQ -----END PGP SIGNATURE----- From solipsis at pitrou.net Tue Apr 27 20:46:46 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Apr 2010 18:46:46 +0000 (UTC) Subject: [Python-Dev] Anyone can do patch reviews References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: Tres Seaver palladion.com> writes: > > This is an excellent set of guidelines. The only drawback I see here is > that the current VCS situation makes doing the review more tedious than > it should be, especially for non-committers. Or maybe the Hg mirrors > are truly up-to-date and working? Last I looked, they were lagging or > unavailable. If you only a review a patch (rather than say maintain and evolve it), there's no point in using hg rather than SVN. However, the mirrors are most of the time alive and up-to-date (sync period is around 5 minutes): http://code.python.org/hg (perhaps you're thinking about http://hg.python.org/, which is an experimental conversion of the SVN repo, not really meant for daily use) Regards Antoine. From guido at python.org Tue Apr 27 20:53:49 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Apr 2010 11:53:49 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100427134618.564e3e3b@heresy> References: <20100427134618.564e3e3b@heresy> Message-ID: On Tue, Apr 27, 2010 at 10:46 AM, Barry Warsaw wrote: > I have two somewhat unrelated thoughts about PEPs. > > * Accepted: header > > When PEP 3147 was accepted, I had a few folks ask that this be recorded in the > PEP by including a link to the BDFL pronouncement email. ?I realized that > there's no formal way to express this in a PEP, and many PEPs in fact don't > record more than the fact that it was accepted. ?I'd like to propose > officially adding an Accepted: header which should include a URL to the email > or other web resource where the PEP is accepted. ?I've come as close as > possible to this (without modifying the supporting scripts or PEP 1) in PEP > 3147: > > ? ?http://www.python.org/dev/peps/pep-3147/ > > I'd be willing to update things if there are no objections. I'd rather not build a single point of failure into the process. Instead of insisting on BDFL pronouncement, the community should switch do something like "last call for objections." There should also be a timeline so that unproductive discussion can't be dragged on forever. > * EOL schedule for older releases. > > We have semi-formal policies for the lifetimes of Python releases, though I'm > not sure this policy is written down in any of the existing informational > PEPs. ?However, we have release schedule PEPs going back to Python 1.6. ?It > seems reasonable to me that we include end-of-life information in those PEPs. > For example, we could state that Python 2.4 is no longer even being maintained > for security, and we could state the projected date that Python 2.6 will go > into security-only maintenance mode. > > I would not mandate that we go back and update all previous PEPs for either of > these ideas. ?We'd adopt them moving forward and allow anyone who's motivated > to backfill information opportunistically. SGTM. -- --Guido van Rossum (python.org/~guido) From tseaver at palladion.com Tue Apr 27 21:23:19 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 27 Apr 2010 15:23:19 -0400 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: > Tres Seaver palladion.com> writes: >> This is an excellent set of guidelines. The only drawback I see here is >> that the current VCS situation makes doing the review more tedious than >> it should be, especially for non-committers. Or maybe the Hg mirrors >> are truly up-to-date and working? Last I looked, they were lagging or >> unavailable. > > If you only a review a patch (rather than say maintain and evolve it), there's > no point in using hg rather than SVN. Hmm, it feels exactly the other way around to me: working with the DVCS tools while reviewiing a patch allows me to be more productive (e.g., using 'bzr shelve' or the equivalent hg subcommand). Making a local branch / clone for each issue also feels more natural than working in a read-only SVN checkout. > However, the mirrors are most of the time alive and up-to-date (sync period is > around 5 minutes): > http://code.python.org/hg That was the URL I was trying to work with: it has been a couple of months since I last tried, however. > (perhaps you're thinking about http://hg.python.org/, which is an experimental > conversion of the SVN repo, not really meant for daily use) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvXOacACgkQ+gerLs4ltQ6ZTwCfQ96RYQ6h/zdMOnFUJU3MkSC1 +o8An2CqK7fbpiCM3gBWZuRReG46xv+U =iWug -----END PGP SIGNATURE----- From ctb at msu.edu Tue Apr 27 21:27:22 2010 From: ctb at msu.edu (C. Titus Brown) Date: Tue, 27 Apr 2010 12:27:22 -0700 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: <20100427192722.GB11994@idyll.org> On Tue, Apr 27, 2010 at 03:23:19PM -0400, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Antoine Pitrou wrote: > > Tres Seaver palladion.com> writes: > >> This is an excellent set of guidelines. The only drawback I see here is > >> that the current VCS situation makes doing the review more tedious than > >> it should be, especially for non-committers. Or maybe the Hg mirrors > >> are truly up-to-date and working? Last I looked, they were lagging or > >> unavailable. > > > > If you only a review a patch (rather than say maintain and evolve it), there's > > no point in using hg rather than SVN. > > Hmm, it feels exactly the other way around to me: working with the DVCS > tools while reviewiing a patch allows me to be more productive (e.g., > using 'bzr shelve' or the equivalent hg subcommand). > > Making a local branch / clone for each issue also feels more natural > than working in a read-only SVN checkout. +1. I find it to be an excellent way to muck around with patches and make my own changes / diffs / etc. for a review process. (Not that I do any Python reviews, note. But it's a great technique in general.) It's also fantastically simple and esay to interact with patches that are branches on someone's bitbucket or github repo; much better than uploading and downloading patch files while in the middle of a discussion. cheers, --titus -- C. Titus Brown, ctb at msu.edu From barry at python.org Tue Apr 27 21:28:45 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Apr 2010 15:28:45 -0400 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: <20100427152845.2c6d95cc@heresy> On Apr 27, 2010, at 02:37 PM, Tres Seaver wrote: >You can always "shelve" the part of the patch which isn't the test: I >do that pretty frequently in the Zope tree, where I am now doing most >development with bzr. Yes definitely. bzr-loom just makes that much easier to manage. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Tue Apr 27 22:38:15 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Apr 2010 22:38:15 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <20100427130305.2C0711FFF13@kimball.webabinitio.net> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> Message-ID: <4BD74B37.6090402@v.loewis.de> R. David Murray wrote: > On Mon, 26 Apr 2010 22:39:39 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: >>>> If possible, I think 'normal' should be the default in the hox or else >>>> there should be some sort of auto replacement. >>> Makes sense to me. >> I have now changed to make 'normal' the default priority for new issues. >> Shall I also set the priority on all past issues to normal which have >> them currently unset? >> >> I would do that "behind" roundup, so that it appears as if the issue was >> already created with that priority. That way, those issues don't appear >> as if they had recent activity. > > +1 > > This will make the default grouping more useful, since now critical > issues will appear at the top, instead of several pages in! Done! Martin From fdrake at acm.org Tue Apr 27 22:45:24 2010 From: fdrake at acm.org (Fred Drake) Date: Tue, 27 Apr 2010 16:45:24 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD74B37.6090402@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> Message-ID: On Tue, Apr 27, 2010 at 4:38 PM, "Martin v. L?wis" wrote: > Done! Thanks, Martin! -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From martin at v.loewis.de Tue Apr 27 23:03:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Apr 2010 23:03:12 +0200 Subject: [Python-Dev] spam on b.p.o In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> Message-ID: <4BD75110.1040404@v.loewis.de> Antoine Pitrou wrote: > Antoine Pitrou pitrou.net> writes: >> Speaking of which, what is the procedure to delete a spam message and remove a >> spamming user? > > Well, for some reason I hadn't seen the "remove button" message... > As for deleting the user, I suppose an admin can do it. For true spam (off-topic links, porn, advertisements, etc), please don't "remove". Instead, people in the "Coordinator" role also have a "mark as spam" button, which causes SpamBayes training of the message, and also makes the message unreadable for anonymous users (including search engines). People posting spam for SEO thus get truly blocked. A mere remove will just detach the message - the message URL in itself remains accessible. In the specific case (msg104314), "remove" was probably the right thing, since it isn't real spam, but just non-sensical. I don't think the user needs to be banned from the tracker. Regards, Martin From martin at v.loewis.de Tue Apr 27 23:06:28 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Apr 2010 23:06:28 +0200 Subject: [Python-Dev] Assigned To field usage In-Reply-To: <20100427124840.4034D1FC6D7@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100427124840.4034D1FC6D7@kimball.webabinitio.net> Message-ID: <4BD751D4.1040804@v.loewis.de> > My perception is that making someone nosy on an issue is preferred to > assigning it to them (allowing them to assign it to themselves if they > think that is appropriate), unless the issue is of higher priority or > someone actively working on the issue really needs the other person's > input in order to move forward. But these are my own rules of thumb, > and a discussion of how best to use this field may be appropriate. I personally think issues should not get assigned unless the person it is being assigned to is known to accept such assignments, e.g. by having asked for them to happen. For example, I would prefer not to be assigned any issues, because I'm unlikely to act on them in the six months - unless, as David says, I'm the *only* person who could move the issue forward in that time. Regards, Martin From martin at v.loewis.de Tue Apr 27 23:09:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Apr 2010 23:09:43 +0200 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> Message-ID: <4BD75297.5020600@v.loewis.de> > Hmm, it feels exactly the other way around to me: working with the DVCS > tools while reviewiing a patch allows me to be more productive (e.g., > using 'bzr shelve' or the equivalent hg subcommand). Just try using Subversion for some time again, and you'll see that it is not difficult at all. Start with a clean sandbox, then apply the patch. If you want to go back to the state without patch, revert the sandbox, if you need it again, apply it again. It's really simple. Regards, Martin From martin at v.loewis.de Tue Apr 27 23:14:55 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 27 Apr 2010 23:14:55 +0200 Subject: [Python-Dev] bug tracker permissions request In-Reply-To: References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> Message-ID: <4BD753CF.6070500@v.loewis.de> > The page doesn't document the Resolution or Status fields. The resolutions are the same as the ones on SourceForge. You only have resolutions on closed issues, and they explain why an issue was closed. If any specific one is unclear in that context, please be more specific. On the status, I hope open/closed are clear, and these are the ones you should be using. Pending is meant as "auto-close in two weeks unless there is follow-up", and its unimplemented. For languishing, click on the label "Status" left of the field. > For the Keywords field, the page only documents the "easy" keyword. So which ones you don't understand? > Also, some of the headings in the page are enclosed in square brackets, > while others are not. It's not clear to me what the brackets are > intended to designate. Not sure what this is referring to - I don't see any square brackets. Regards, Martin From solipsis at pitrou.net Tue Apr 27 23:14:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Apr 2010 21:14:41 +0000 (UTC) Subject: [Python-Dev] spam on b.p.o References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> <4BD75110.1040404@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > In the specific case (msg104314), "remove" was probably the right thing, > since it isn't real spam, but just non-sensical. I don't think the user > needs to be banned from the tracker. The message was a copy of a previous message by someone else, with an additional HTML link in the middle. The target of that link was clearly the kind that would pay to increase its Google rank through whatever means (bogus diploma stuff, IIRC). Regards Antoine. From tseaver at palladion.com Tue Apr 27 23:21:39 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 27 Apr 2010 17:21:39 -0400 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: <4BD75297.5020600@v.loewis.de> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <4BD75297.5020600@v.loewis.de> Message-ID: <4BD75563.1020406@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >> Hmm, it feels exactly the other way around to me: working with the DVCS >> tools while reviewiing a patch allows me to be more productive (e.g., >> using 'bzr shelve' or the equivalent hg subcommand). > > Just try using Subversion for some time again, and you'll see that it is > not difficult at all. Start with a clean sandbox, then apply the patch. > If you want to go back to the state without patch, revert the sandbox, > if you need it again, apply it again. It's really simple. I use Subversion daily as a committer on multiple projects. Keeping multiple readon-only checkouts around to evaluate patches is much clunkier than maintaining DVCS branches for the same purpose. As a non-committer, what I would miss most is the ability to make local commits. Features like "shelve", "log -p", etc., are aslo extremely useful when analyzing a patch. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvXVV8ACgkQ+gerLs4ltQ7JqwCfVZPkm8/3XUuuzpm/N+08x2RI KWYAn004cLJS3poYZ/4BvSFOGzMpwNuC =j3lH -----END PGP SIGNATURE----- From martin at v.loewis.de Tue Apr 27 23:26:20 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 27 Apr 2010 23:26:20 +0200 Subject: [Python-Dev] spam on b.p.o In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <20100426061223.GA17467@tummy.com> <20100426093153.GA61243@nexus.in-nomine.org> <4BD75110.1040404@v.loewis.de> Message-ID: <4BD7567C.7050400@v.loewis.de> > The message was a copy of a previous message by someone else, with an additional > HTML link in the middle. The target of that link was clearly the kind that would > pay to increase its Google rank through whatever means (bogus diploma stuff, > IIRC). Ah, I missed that. I've marked it as spam now. Regards, Martin From ncoghlan at gmail.com Tue Apr 27 23:47:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Apr 2010 07:47:52 +1000 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: <4BD75B88.1060709@gmail.com> Ezio Melotti wrote: > When I'm writing the patch it's usually easy, I write the tests, see > that they fail, write the fix, see that they pass. > When I'm reviewing the patch, I apply the patch, see that the tests > pass, svn revert the fix, check that they fail. > Most of the patches affect just a couple of files, so applying the whole > patch and then revert is usually trivial and probably easier than having > to deal with two separate files for patch and tests. This would be pretty close to my typical workflow as well (*looks at list of assigned bugs that hasn't moved in weeks* well, it is the workflow when I actually *do* some Python coding...) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Apr 27 23:55:32 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Apr 2010 07:55:32 +1000 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> Message-ID: <4BD75D54.8050307@gmail.com> Guido van Rossum wrote: >> When PEP 3147 was accepted, I had a few folks ask that this be recorded in the >> PEP by including a link to the BDFL pronouncement email. I realized that >> there's no formal way to express this in a PEP, and many PEPs in fact don't >> record more than the fact that it was accepted. I'd like to propose >> officially adding an Accepted: header which should include a URL to the email >> or other web resource where the PEP is accepted. I've come as close as >> possible to this (without modifying the supporting scripts or PEP 1) in PEP >> 3147: >> >> http://www.python.org/dev/peps/pep-3147/ >> >> I'd be willing to update things if there are no objections. > > I'd rather not build a single point of failure into the process. > Instead of insisting on BDFL pronouncement, the community should > switch do something like "last call for objections." There should also > be a timeline so that unproductive discussion can't be dragged on > forever. I believe the more important part of Barry's suggested change here is requiring a link to the archived message (usually from python-dev) where the PEP was accepted (be it directly by you as BDFL, or by consensus from a "sufficient" number of core developers). This will likely also help with reminding people to announce on python-dev when PEPs are accepted by consensus (or by you) somewhere like PyCon or a sprint. >> I would not mandate that we go back and update all previous PEPs for either of >> these ideas. We'd adopt them moving forward and allow anyone who's motivated >> to backfill information opportunistically. > > SGTM. +1 to both ideas from me, too. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jcea at jcea.es Wed Apr 28 00:43:08 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 28 Apr 2010 00:43:08 +0200 Subject: [Python-Dev] Anyone can do patch reviews In-Reply-To: <20100427111651.738d2381@heresy> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> <20100427111651.738d2381@heresy> Message-ID: <4BD7687C.3080001@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 27/04/10 17:16, Barry Warsaw wrote: > It would be kind of cool if there were some best practices for running said > unittest both with and without the patch enabled. Kind of like using #ifdefs > in C but without all the commenting-out-commenting-in error proneness. I > guess you could do something like Mercurial queues are very useful here. You apply/deapply patches with a single command: I am using python SVN mercurial mirror and MQ (Mercurial Queues) for development, waiting for the "real thing" (Mercurial native working). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS9dofJlgi5GaxT1NAQIyAQP/avvYJxxlY4lr58nHbjsuoROz1rQi7RrR qd8G3grsS9NXlYbygw0rERJyg9UgjDhJrZbwYEPGJkxTIUd/Vcnw/fIB6J+xuLlY sRnmh0P6ILOFTHYoZZZ/hxtfdMiZxqiMHO3Pfs8uBc5bGC0f23cqiTOFY0+ze7mU 3vUIcljhuRE= =oyQb -----END PGP SIGNATURE----- From steve at pearwood.info Wed Apr 28 02:46:37 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Apr 2010 10:46:37 +1000 Subject: [Python-Dev] =?iso-8859-1?q?Enhanced_tracker_privileges_for_dange?= =?iso-8859-1?q?rjim_to_do=09triage=2E?= In-Reply-To: <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <201004281046.37939.steve@pearwood.info> On Tue, 27 Apr 2010 12:59:26 pm Stephen J. Turnbull wrote: > Steve Holden writes: > > Yes, in the last year in particular there has been some excellent > > effort of maintaining the issue tracker content. But the question > > still remains - who are we worried about offending? > > In this thread, we did worry about offending Sean and dangerjim. Now > that Sean has commented, I don't think anybody is worrying about > offending anybody; there is an understanding that there's a process > issue to be resolved. The question is how best to build the > community. > > There are two camps, the quantity camp ("low cost of contribution > means more contributors, and that's good"), and the quality camp > ("more interaction within the community, especially of experienced > developers with newcomers, means more effective contributors and > that's good"). I'm not sure that is a relevant division between the two camps. I think both sides recognise the need to increase the number of contributors without compromising on their quality. I haven't heard anyone say that we have enough people to do the work that needs doing, or that we should take any warm body who can spell PC. As I see it, the two camps are divided purely on the question of how to get increased privileges. Both sides agree that merit is a requirement, but the disagreement is on how to prove you have such merit. One side insists that the only way to prove merit is to go through a period of untrusted, unprivileged contributions, with no exceptions. One argument for this is that unless we treat everyone identically, some would-be contributors will see the process as nepotism, or otherwise be offended that they didn't get the "special treatment". I believe this misses the point that we *don't* treat people identically, nor can we in a meritocracy. People are treated differently according to not just the quality of their patches, but also the speed at which they submit them, and even more importantly, their ability to gain recognition for their merit. It's not enough to be good at what you do, people have to know it. Ten high-quality patches for high-profile bugs in a week may get you enhanced privileges, while thirty high-quality patches for low-profile bugs in six years might not, simply because nobody notices you. The other side says that a second way of proving merit is through reputation and having a trusted contributor vouch for you, and offer to mentor you. The major difference here is that it's not mandatory to prove your merit to the entire community, but sufficient to prove it to a single trusted member of the community who is willing to stake his own reputation on your ability to perform. Suppose the PSF were to hire somebody specifically to work on patches in the tracker, chances are they would be somebody well-known to the community. But suppose they weren't -- suppose the hirer read dozens of CVs, performed interviews, and determined that the best person for the job was somebody utterly unknown to the community, somebody who had been working quietly away doing brilliant things in Python but with no public profile, and offered her the job of committing patches. Would she get elevated privileges immediately? [...] > *By definition*, a community is not diverse in the most fundamental > sense. I think you're using a definition of community that doesn't appear in any dictionary I'm aware of, nor do I understand what you mean by "most fundamental sense" of diverse. Talking about diversity within a single community is not an oxymoron. > As long as Pythonicity is important to Python, there is > danger as well as opportunity in more rapid influx of newcomers. This at least is true. I can't dispute that. -- Steven D'Aprano From tjreedy at udel.edu Wed Apr 28 02:53:49 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Apr 2010 20:53:49 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD74B37.6090402@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> Message-ID: On 4/27/2010 4:38 PM, "Martin v. L?wis" wrote: > Done! When I open http://bugs.python.org/issue?@template=item priority is (still) set at no selection. Is this my local cache (which I do not know how to clear in FF) or is 'normal' filled in after submission? I do verify that searching for any issue with priority 'not selected' returns an empty list. Terry Jan Reedy From martin at v.loewis.de Wed Apr 28 08:43:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 28 Apr 2010 08:43:44 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> Message-ID: <4BD7D920.8020200@v.loewis.de> > When I open http://bugs.python.org/issue?@template=item > priority is (still) set at no selection. Is this my local cache (which I > do not know how to clear in FF) or is 'normal' filled in after submission? It is filled in after submission. Regards, Martin From dirkjan at ochtman.nl Wed Apr 28 09:22:01 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 28 Apr 2010 09:22:01 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BD75D54.8050307@gmail.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> Message-ID: On Tue, Apr 27, 2010 at 23:55, Nick Coghlan wrote: > I believe the more important part of Barry's suggested change here is > requiring a link to the archived message (usually from python-dev) where > the PEP was accepted (be it directly by you as BDFL, or by consensus > from a "sufficient" number of core developers). This will likely also > help with reminding people to announce on python-dev when PEPs are > accepted by consensus (or by you) somewhere like PyCon or a sprint. Though maybe it should be called Conclusion instead of Accepted and used for Rejected PEPs, as well? Cheers, Dirkjan From stephen at xemacs.org Wed Apr 28 10:16:48 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Apr 2010 17:16:48 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dange rjim to do triage. In-Reply-To: <201004281046.37939.steve@pearwood.info> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> Message-ID: <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > As I see it, the two camps are divided purely on the question of how to > get increased privileges. As I see it, the division is over what constitutes merit, and how it is created or improved. > Both sides agree that merit is a requirement, but the disagreement > is on how to prove you have such merit. I disagree vehemently with that characterization of my position (and I strongly suspect David would, too). The primary argument of the "quality" camp as I see it is that the familiarization period *creates* value, both in terms of training ("merit" for the job) and interpersonal relationships ("building community"). Thus it is a *net benefit*, not a *net cost*. AFAICS, the "quantity" camp sees it as a nearly pure loss, simply slowing down inflow of preexisting "merit" (and perhaps discouraging it entirely). > > *By definition*, a community is not diverse in the most fundamental > > sense. > > I think you're using a definition of community that doesn't appear in > any dictionary I'm aware of, nor do I understand what you mean by "most > fundamental sense" of diverse. Talking about diversity within a single > community is not an oxymoron. Where did I write "oxymoron"? The grammar was a bit awkward, but my point is simple: the root of the word "community" is *common*. Therefore it makes sense to bring in newcomers via a process which accustoms them to the commonality, of whatever degree, the community is based on. From stephen at xemacs.org Wed Apr 28 11:22:02 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Apr 2010 18:22:02 +0900 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <4BD7D920.8020200@v.loewis.de> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> <4BD7D920.8020200@v.loewis.de> Message-ID: <87633c7wzp.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > > When I open http://bugs.python.org/issue?@template=item > > priority is (still) set at no selection. Is this my local cache (which I > > do not know how to clear in FF) or is 'normal' filled in after submission? > > It is filled in after submission. You still want the auditor to make sure, but it is possible to set the default in the page template. Would you like a patch for that for consideration? If so, are there other elements it might be useful to have defaults for? (I would submit each as a separate patch.) From steve at pearwood.info Wed Apr 28 13:01:23 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Apr 2010 21:01:23 +1000 Subject: [Python-Dev] =?iso-8859-1?q?Enhanced_tracker_privileges_for_dange?= =?iso-8859-1?q?_rjim_to_do=09triage=2E?= In-Reply-To: <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <201004281046.37939.steve@pearwood.info> <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <201004282101.24581.steve@pearwood.info> On Wed, 28 Apr 2010 06:16:48 pm Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > As I see it, the two camps are divided purely on the question of > > how to get increased privileges. > > As I see it, the division is over what constitutes merit, and how it > is created or improved. > > > Both sides agree that merit is a requirement, but the disagreement > > is on how to prove you have such merit. > > I disagree vehemently with that characterization of my position (and > I strongly suspect David would, too). The primary argument of the > "quality" camp as I see it is that the familiarization period > *creates* value, both in terms of training ("merit" for the job) and > interpersonal relationships ("building community"). Thus it is a > *net benefit*, not a *net cost*. AFAICS, the "quantity" camp sees it > as a nearly pure loss, simply slowing down inflow of preexisting > "merit" (and perhaps discouraging it entirely). Thank you for clarifying your position. I disagree with it, but at least now I understand it better. -- Steven D'Aprano From rdmurray at bitdance.com Wed Apr 28 13:26:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 28 Apr 2010 07:26:47 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100428112647.A2D631FF378@kimball.webabinitio.net> On Wed, 28 Apr 2010 17:16:48 +0900, "Stephen J. Turnbull" wrote: > Steven D'Aprano writes: > > > As I see it, the two camps are divided purely on the question of how to > > get increased privileges. > > As I see it, the division is over what constitutes merit, and how it > is created or improved. > > > Both sides agree that merit is a requirement, but the disagreement > > is on how to prove you have such merit. > > I disagree vehemently with that characterization of my position (and > I strongly suspect David would, too). The primary argument of the > "quality" camp as I see it is that the familiarization period > *creates* value, both in terms of training ("merit" for the job) and > interpersonal relationships ("building community"). Thus it is a *net > benefit*, not a *net cost*. AFAICS, the "quantity" camp sees it as a > nearly pure loss, simply slowing down inflow of preexisting "merit" > (and perhaps discouraging it entirely). Except for the "vehemently" part, I think that's an accurate summary of my position. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Wed Apr 28 13:29:41 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 28 Apr 2010 07:29:41 -0400 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <87633c7wzp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> <4BD7D920.8020200@v.loewis.de> <87633c7wzp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100428112941.C62691FFB4F@kimball.webabinitio.net> On Wed, 28 Apr 2010 18:22:02 +0900, "Stephen J. Turnbull" wrote: > You still want the auditor to make sure, but it is possible to set the > default in the page template. Would you like a patch for that for > consideration? If so, are there other elements it might be useful to > have defaults for? (I would submit each as a separate patch.) As far as I can see none of the other fields should have a default value. -- R. David Murray www.bitdance.com From amk at amk.ca Wed Apr 28 14:05:48 2010 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 28 Apr 2010 08:05:48 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do?triage. In-Reply-To: <201004281046.37939.steve@pearwood.info> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> Message-ID: <20100428120548.GA2487@amk-desktop.matrixgroup.net> On Wed, Apr 28, 2010 at 10:46:37AM +1000, Steven D'Aprano wrote: > their ability to gain recognition for their merit. It's not enough to > be good at what you do, people have to know it. Ten high-quality > patches for high-profile bugs in a week may get you enhanced > privileges, while thirty high-quality patches for low-profile bugs in > six years might not, simply because nobody notices you. True. This makes me wonder if we should be data-mining the tracker information to look for significant contributors that no one has noticed. --amk From solipsis at pitrou.net Wed Apr 28 14:18:04 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Apr 2010 12:18:04 +0000 (UTC) Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do?triage. References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> <20100428120548.GA2487@amk-desktop.matrixgroup.net> Message-ID: A.M. Kuchling amk.ca> writes: > > True. This makes me wonder if we should be data-mining the tracker > information to look for significant contributors that no one has > noticed. I'd say that we usually notice them, since we process their patches or read their reviews and comments. Unless perhaps they're called "John Smith" or "Jean Dupont" (if French). Regards Antoine. From stephen at xemacs.org Wed Apr 28 16:11:15 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Apr 2010 23:11:15 +0900 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do?triage. In-Reply-To: <20100428120548.GA2487@amk-desktop.matrixgroup.net> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> <20100428120548.GA2487@amk-desktop.matrixgroup.net> Message-ID: <871vdz8y64.fsf@uwakimon.sk.tsukuba.ac.jp> A.M. Kuchling writes: > True. This makes me wonder if we should be data-mining the tracker > information to look for significant contributors that no one has > noticed. It's an interesting idea. But I've done something similar to this, ad hoc[1], a few times in XEmacs, and it has never worked out. That's not to say it never will, but I found that people with a long-term low-level interest generally have a reason for being that way. Footnotes: [1] In looking through logs, I see a person whose name I recognize from other commits associated with a currently problematic area, and get in touch. They've been happy to give what advice they can on the current issue, but then gracefully decline more regular involvement. From martin at v.loewis.de Wed Apr 28 19:18:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 28 Apr 2010 19:18:13 +0200 Subject: [Python-Dev] Make 'normal' the default tracker priority level In-Reply-To: <87633c7wzp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <78960cb245bcaedc307bd083093b2c19.squirrel@mail.trueblade.com> <4BD5FA0B.8040504@v.loewis.de> <20100427130305.2C0711FFF13@kimball.webabinitio.net> <4BD74B37.6090402@v.loewis.de> <4BD7D920.8020200@v.loewis.de> <87633c7wzp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BD86DD5.1040801@v.loewis.de> > You still want the auditor to make sure, but it is possible to set the > default in the page template. Would you like a patch for that for > consideration? If so, are there other elements it might be useful to > have defaults for? (I would submit each as a separate patch.) Feel free to submit patches. I don't want to spend more time on this soon, though. Regards, Martin From amk at amk.ca Thu Apr 29 01:54:34 2010 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 28 Apr 2010 19:54:34 -0400 Subject: [Python-Dev] Anyone can do patch reviews (was: Enhanced tracker privileges...) In-Reply-To: <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> References: <4BD4A69A.4080507@tummy.com> <201004270522.38521.steve@pearwood.info> <201004271115.50329.steve@pearwood.info> <20100427133814.785331FFC99@kimball.webabinitio.net> <20100427144019.2091.2017478396.divmod.xquotient.2@localhost.localdomain> Message-ID: <20100428235434.GA25602@andrew-kuchlings-macbook.local> On Tue, Apr 27, 2010 at 02:40:19PM -0000, exarkun at twistedmatrix.com wrote: > This list would make a good addition to one of the cpython development > pages. If potential contributors could find this information, then > they'd be much more likely to participate by doing reviews. If anyone wants to write the updated text, the following command will make an anonymous read-only checkout of the /dev/ pages: svn ls https://svn.python.org/www/trunk/beta.python.org/build/data/dev I can apply a patch. --amk From ncoghlan at gmail.com Thu Apr 29 12:16:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Apr 2010 20:16:14 +1000 Subject: [Python-Dev] Frequency of the dev docs autobuild Message-ID: <4BD95C6E.9080508@gmail.com> Does the online dev version of the docs build in response to docs checkins, or just once a day? (And is that written down somewhere and I've just forgotten where to look...) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Thu Apr 29 13:40:47 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Apr 2010 07:40:47 -0400 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <4BD95C6E.9080508@gmail.com> References: <4BD95C6E.9080508@gmail.com> Message-ID: <20100429114047.C031C20059D@kimball.webabinitio.net> On Thu, 29 Apr 2010 20:16:14 +1000, Nick Coghlan wrote: > Does the online dev version of the docs build in response to docs > checkins, or just once a day? I believe it does it once a day. Georg recently changed how this is done, so we should get official word from him. > (And is that written down somewhere and I've just forgotten where to > look...) I don't think so, but it should be. -- R. David Murray www.bitdance.com From janssen at parc.com Thu Apr 29 16:39:22 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 29 Apr 2010 07:39:22 PDT Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> Message-ID: <96960.1272551962@parc.com> Ned Deily wrote: > In article <19399.11323.946604.992452 at montanaro.dyndns.org>, > skip at pobox.com wrote: > > > Ned> Any idea what type of machine it is and where it is currently > > Ned> located? > > > > I seem to recall it is/was a G4 XServe. My guess as to location would be at > > xs4all.nl. > > If it were working that could be of use. It would not be able to run OS > X 10.6 but having a 10.5 system PPC system as a buildbot would certainly > be useful; it should be fine for the default installer configuration > builds. (Alas, I don't expect to be anywhere in the vicinity in the > foreseeable future.) I've identified 4 G4/G5 machines at PARC I can set up as build slaves on either Tiger or Leopard, and the first one of those is up (G4/Tiger) and running. Can someone (Ned?) take a look at the output and see if the errors are currently expected, or if I need to do more config/install work? I'm using Xcode 2.5. If it's good, I'll go to work on more. The build slave name is "parc-tiger-1". Thanks! Bill From solipsis at pitrou.net Thu Apr 29 17:05:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Apr 2010 15:05:34 +0000 (UTC) Subject: [Python-Dev] Tiger buildbots References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> Message-ID: Le Thu, 29 Apr 2010 07:39:22 -0700, Bill Janssen a ?crit?: > > I've identified 4 G4/G5 machines at PARC I can set up as build slaves on > either Tiger or Leopard, and the first one of those is up (G4/Tiger) and > running. Can someone (Ned?) take a look at the output and see if the > errors are currently expected, or if I need to do more config/install > work? I'm using Xcode 2.5. If it's good, I'll go to work on more. No failures are currently expected, but the other Tiger buildbot is seeing the same ones as yours. In any case, somebody needs to investigate why these tests fail under Tiger, and I suppose the buildbot owners are in the best position to do so. Regards Antoine. From ronaldoussoren at mac.com Thu Apr 29 17:13:39 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 29 Apr 2010 17:13:39 +0200 Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <96960.1272551962@parc.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> Message-ID: <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> On 29 Apr, 2010, at 16:39, Bill Janssen wrote: > Ned Deily wrote: > >> In article <19399.11323.946604.992452 at montanaro.dyndns.org>, >> skip at pobox.com wrote: >> >>> Ned> Any idea what type of machine it is and where it is currently >>> Ned> located? >>> >>> I seem to recall it is/was a G4 XServe. My guess as to location would be at >>> xs4all.nl. >> >> If it were working that could be of use. It would not be able to run OS >> X 10.6 but having a 10.5 system PPC system as a buildbot would certainly >> be useful; it should be fine for the default installer configuration >> builds. (Alas, I don't expect to be anywhere in the vicinity in the >> foreseeable future.) > > I've identified 4 G4/G5 machines at PARC I can set up as build slaves > on either Tiger or Leopard, and the first one of those is up (G4/Tiger) > and running. Can someone (Ned?) take a look at the output and see if > the errors are currently expected, or if I need to do more > config/install work? I'm using Xcode 2.5. If it's good, I'll go to > work on more. > > The build slave name is "parc-tiger-1". As Antoine noted the test failures are unexpected, could you check if the tests pass if you do the build and testrun manually? IMHO it would be better to do a framework build for at least some of the OSX buildbots because that's what is in the binary installer for OSX. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From janssen at parc.com Thu Apr 29 18:40:48 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 29 Apr 2010 09:40:48 PDT Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> Message-ID: <136.1272559248@parc.com> Ronald Oussoren wrote: > As Antoine noted the test failures are unexpected, could you check if > the tests pass if you do the build and testrun manually? I'll have to fiddle with where it "is" on our network to do that. I'll take it down and move it to different subnet. What about this error? /bin/sh: line 1: pybuildbot.identify: command not found It looks like a configuration issue, or maybe a bug in pybuildbot. What about readline? Darwin doesn't have it (or rather, it has a different one). Why is that 'skip' unexpected? > IMHO it would be better to do a framework build for at least some of > the OSX buildbots because that's what is in the binary installer for > OSX. Yes, that sounds good to me, too. But how do we make that a standard test so that appropriately enabled buildbot slaves will do it automatically? Something that gets configured in the build master? Bill From fuzzyman at voidspace.org.uk Thu Apr 29 19:35:16 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 29 Apr 2010 18:35:16 +0100 Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> Message-ID: <4BD9C354.2010902@voidspace.org.uk> On 29/04/2010 16:13, Ronald Oussoren wrote: > On 29 Apr, 2010, at 16:39, Bill Janssen wrote: > > >> Ned Deily wrote: >> >> >>> In article<19399.11323.946604.992452 at montanaro.dyndns.org>, >>> skip at pobox.com wrote: >>> >>> >>>> Ned> Any idea what type of machine it is and where it is currently >>>> Ned> located? >>>> >>>> I seem to recall it is/was a G4 XServe. My guess as to location would be at >>>> xs4all.nl. >>>> >>> If it were working that could be of use. It would not be able to run OS >>> X 10.6 but having a 10.5 system PPC system as a buildbot would certainly >>> be useful; it should be fine for the default installer configuration >>> builds. (Alas, I don't expect to be anywhere in the vicinity in the >>> foreseeable future.) >>> >> I've identified 4 G4/G5 machines at PARC I can set up as build slaves >> on either Tiger or Leopard, and the first one of those is up (G4/Tiger) >> and running. Can someone (Ned?) take a look at the output and see if >> the errors are currently expected, or if I need to do more >> config/install work? I'm using Xcode 2.5. If it's good, I'll go to >> work on more. >> >> The build slave name is "parc-tiger-1". >> > As Antoine noted the test failures are unexpected, could you check if the tests pass if you do the build and testrun manually? > > Well - I have nine failing tests on trunk for Mac OS X with Snow Leopard. 9 tests failed: test_cmd_line test_imp test_import test_posix test_pydoc test_runpy test_urllib2 test_urllib2_localnet test_warnings I believe that Victor has ensured that the buildbot failures have open issues against, I'm just going to check that the failures I see are the same. Michael > IMHO it would be better to do a framework build for at least some of the OSX buildbots because that's what is in the binary installer for OSX. > > Ronald > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Apr 29 20:11:57 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 Apr 2010 14:11:57 -0400 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: <4BD753CF.6070500@v.loewis.de> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> Message-ID: On 4/27/2010 5:14 PM, "Martin v. L?wis" wrote: >> The page doesn't document the Resolution or Status fields. > > The resolutions are the same as the ones on SourceForge. They no longer have to be (see below). In any case, the meanings should be independently documented here for people not currently using SF. > You only have resolutions on closed issues, Only for 'closed' or also for 'pending' and 'languishing'? If pending were implemented to mean 'auto close in x days', then Resolution would need to be set when Status is set to 'pending'. 'Languishing' is new (its first use was two months ago). Is it a form of 'open' or of 'closed'? > and they explain why an issue was closed. This needs to be explicitly stated in the doc if you want it followed. In response to my repeating this on the tracker, bugs.python.org/issue7511 R. David Murray said (in part) "Some committers have been using 'resolution: accepted' as a way to mark issues that are definitely going to be fixed/enhancement added." This is a different and therefore confusing use of the field and strikes me as mostly redundant with the newer Stage field with a value of Needs patch or later. > If any specific one is unclear in that context, please be more specific. Tarek responded to David's full message "Yes I used 'accepted' to mark this issue as something I am working on." This usage overlaps with 'assigned to'. And he continued "I am confused about the resolution flag, I think it is a bit overlapping with the status flag, and overcomplex." This *is* overlap Resolution set means Status not open. I agree with the over-complex part. There was a discussion here some time ago on the fine distinctions, but I am again fuzzy on 'accepted' versus 'fixed', 'later' versus 'postponed', and 'invalid' versus 'works for me'. I have no idea when 'remind' would be used (it has only been used 8 times). > For languishing, click on the label "Status" left of the field. I never noticed this. there are also pop-up boxed for Stage, Priority, and Keywords. There should also be a pop-up for Resolution. And the pop-up tables should also be in the doc. >> For the Keywords field, the page only documents the "easy" keyword. There are all documented in the pop-up box, which should be duplicated in the doc. Terry Jan Reedy From janssen at parc.com Thu Apr 29 20:28:47 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 29 Apr 2010 11:28:47 PDT Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <4BD9C354.2010902@voidspace.org.uk> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> Message-ID: <22670.1272565727@parc.com> Michael Foord wrote: > Well - I have nine failing tests on trunk for Mac OS X with Snow Leopard. > > 9 tests failed: > test_cmd_line test_imp test_import test_posix test_pydoc > test_runpy test_urllib2 test_urllib2_localnet test_warnings > > I believe that Victor has ensured that the buildbot failures have open > issues against, I'm just going to check that the failures I see are > the same. Just built on Intel/Leopard with the trunk (almost vanilla, except for _ssl.c) and tested. Here's what I configured: ./configure --prefix=/local/python/trunk --disable-universalsdk --disable-framework --disable-toolbox-glue --with-pydebug And here's the results with "make test": 342 tests OK. 6 tests failed: test_argparse test_grp test_posix test_pwd test_socket_ssl test_urllib2_localnet 37 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_epoll test_gdb test_gdbm test_gl test_imgfile test_largefile test_linuxaudiodev test_macos test_macostools test_ossaudiodev test_readline test_scriptpackages test_smtpnet test_socketserver test_startfile test_sunaudiodev test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 8 skips unexpected on darwin: test_aepack test_applesingle test_gdb test_macos test_macostools test_readline test_scriptpackages test_ttk_guionly Why is the skip of "test_readline" unexpected on darwin? The readline on Darwin isn't what Python wants. Bill From rdmurray at bitdance.com Thu Apr 29 20:47:33 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Apr 2010 14:47:33 -0400 Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <22670.1272565727@parc.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> Message-ID: <20100429184733.3009420087A@kimball.webabinitio.net> On Thu, 29 Apr 2010 11:28:47 -0700, Bill Janssen wrote: > 8 skips unexpected on darwin: > test_aepack test_applesingle test_gdb test_macos test_macostools > test_readline test_scriptpackages test_ttk_guionly > > Why is the skip of "test_readline" unexpected on darwin? The readline > on Darwin isn't what Python wants. I think someone fixed readline to work with Darwin's readline, at least in theory. -- R. David Murray www.bitdance.com From billjordan121 at yahoo.com Thu Apr 29 19:58:15 2010 From: billjordan121 at yahoo.com (Bill Jordan) Date: Thu, 29 Apr 2010 10:58:15 -0700 (PDT) Subject: [Python-Dev] array matching Message-ID: <83538.68421.qm@web114307.mail.gq1.yahoo.com> Hey guys, I am?sorry if this is not the right list to post some questions. I have a simple question please and would appreciate some answers as I am new to Python. I have 2 D array: test = [[A,1],[B,2],[A,3][B,4]] I want to arrang this array in different arrays so each one will have what is attached to. For example I want the output: A =[1,3] and B=[2,4] Thanks, Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Thu Apr 29 21:22:38 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Thu, 29 Apr 2010 23:22:38 +0400 Subject: [Python-Dev] array matching In-Reply-To: <83538.68421.qm@web114307.mail.gq1.yahoo.com> References: <83538.68421.qm@web114307.mail.gq1.yahoo.com> Message-ID: <20100429192238.GA7923@phd.pp.ru> Hello. We'are sorry but we cannot help you. This mailing list is to work on developing Python (fixing bugs and adding new features to Python itself); if you're having problems using Python, please find another forum. Probably python-list (comp.lang.python) news group/mailing list is the best place. See http://www.python.org/community/lists/ for other lists/news groups/fora. Thank you for understanding. On Thu, Apr 29, 2010 at 10:58:15AM -0700, Bill Jordan wrote: > I am?sorry if this is not the right list to post some questions. I have a simple question please and would appreciate some answers as I am new to Python. > > I have 2 D array: test = [[A,1],[B,2],[A,3][B,4]] > I want to arrang this array in different arrays so each one will have what is attached to. For example I want the output: > > A =[1,3] and B=[2,4] > > Thanks, > Bill Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Thu Apr 29 21:24:15 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 Apr 2010 15:24:15 -0400 Subject: [Python-Dev] array matching In-Reply-To: <83538.68421.qm@web114307.mail.gq1.yahoo.com> References: <83538.68421.qm@web114307.mail.gq1.yahoo.com> Message-ID: <4BD9DCDF.2020000@udel.edu> On 4/29/2010 1:58 PM, Bill Jordan wrote: > I am sorry if this is not the right list to post some questions. You cross-posted to 4. Not good as answers will go to all 4 if responders are not careful. Gmane.comp.python.devel is only for the development of the next versions of CPython and mainly for developers. Please to not send general questions here. Terry Jan Reedy From rdmurray at bitdance.com Thu Apr 29 22:05:33 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Apr 2010 16:05:33 -0400 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> Message-ID: <20100429200533.18CE720087A@kimball.webabinitio.net> On Thu, 29 Apr 2010 14:11:57 -0400, Terry Reedy wrote: > On 4/27/2010 5:14 PM, "Martin v. Loewis" wrote: > > You only have resolutions on closed issues, > > Only for 'closed' or also for 'pending' and 'languishing'? > If pending were implemented to mean 'auto close in x days', then > Resolution would need to be set when Status is set to 'pending'. Definitely. Resolution should always be set for Pending, IMO. > 'Languishing' is new (its first use was two months ago). Is it a form of > 'open' or of 'closed'? Open. > > and they explain why an issue was closed. > > This needs to be explicitly stated in the doc if you want it followed. > > In response to my repeating this on the tracker, > bugs.python.org/issue7511 > R. David Murray said (in part) > "Some committers have been using 'resolution: accepted' as a way to mark > issues that are definitely going to be fixed/enhancement added." > > This is a different and therefore confusing use of the field and strikes > me as mostly redundant with the newer Stage field with a value of Needs > patch or later. Well, where it gets ambiguous is that the stage can get set to unit test or patch needed because that's what stage the bug report is in, without anyone having made a decision as to whether or not the patch will actually get applied if the patch is completed. So some way of indicating that the bug/enhancement is 'accepted' and will be applied as soon as it passes muster seems like a good idea, as opposed to issues where the evolution of the patch is part of the decision making process as to whether to accept it or not. I agree that having that be the 'resolution' field is logically and pedagogically incorrect ;) I've made a proposal (discussed with some of the other triage people) that would clarify this (and perhaps ultimately lead to the elimination of the stage field, although I don't mention that idea on the wiki), at: http://wiki.python.org/moin/DesiredTrackerFeatures I believe Ezio will be looking at this this summer. > > If any specific one is unclear in that context, please be more specific. [...] > over-complex part. There was a discussion here some time ago on the fine > distinctions, but I am again fuzzy on 'accepted' versus 'fixed', 'later' > versus 'postponed', and 'invalid' versus 'works for me'. I have no idea > when 'remind' would be used (it has only been used 8 times). In my mind some of these are clear (at least when using the field as it is labeled: for 'resolution'), but clearly the docs need help. 'accepted' would be for an enhancement (committing an enhancement doesn't 'fix' anything), while 'fixed' means the bug no longer exists. 'invalid' means that the bug report is just wrong, it's not a bug, while 'works for me' usually means that we can't reproduce the problem. I agree that some bugs that get closed 'works for me' really should be closed using another resolution. I doubt the reverse is true. I suspect 'works for me' gets used sometimes instead of invalid because the closer isn't sure that everyone on the dev teams shares their opinion that the bug is invalid or; more often, in cases where "won't fix" would be more appropriate. 'Accepted' and 'fixed' could be replaced by the single resolution 'committed'. That will eliminate the ambiguity of using 'accepted' as something other than a resolution. 'Works for me' should probably be dropped and replaced by "can't reproduce". 'later', 'remind', and 'postponed' all seem too close to be useful distinctions, and all seem pretty much equivalent to 'languishing'. IMO only one of the four should be kept. I prefer 'languishing' because it shows up as an issue count in the weekly summary, and it seems to me to me that this marking for an issue is more of a status than a resolution. (That is, 'later', 'remind', and 'postponed' are *not* resolutions, they are procrastinations, which 'languishing' makes explicit.) > > For languishing, click on the label "Status" left of the field. > > and Keywords. There should also be a pop-up for Resolution. > And the pop-up tables should also be in the doc. It would be great if you would open an issue for these suggestions in the meta-tracker, so that they don't get lost. If there is no objection to the resolution changes I suggest above, I can do those. I believe issues closed with those resolutions would keep them, and that there is a way for someone to mine for them if they ever wanted to go through and reevaluate them. I could optionally change all 'accepted' and 'fixed' into 'committed' since that change is pretty unambiguous, but the others I think should just be left unchanged on the existing issues unless someone feels moved to change a given issue by hand. This would leave us with the following list of resolutions: committed out of date duplicate can't reproduce invalid won't fix rejected The only remaining redundancy is "won't fix" versus "rejected", but I can't think of single word that would cover both cases (we refuse to fix a bug for various reasons, but we reject an enhancement request). -- R. David Murray www.bitdance.com From solipsis at pitrou.net Thu Apr 29 22:08:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Apr 2010 20:08:44 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?please_take_a_look_at_buildbot_result_=5Bw?= =?utf-8?q?as=3A_Broken=09link_to_download_=28Mac_OS_X=29=5D?= References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> <20100429184733.3009420087A@kimball.webabinitio.net> Message-ID: R. David Murray bitdance.com> writes: > > On Thu, 29 Apr 2010 11:28:47 -0700, Bill Janssen wrote: > > 8 skips unexpected on darwin: > > test_aepack test_applesingle test_gdb test_macos test_macostools > > test_readline test_scriptpackages test_ttk_guionly > > > > Why is the skip of "test_readline" unexpected on darwin? The readline > > on Darwin isn't what Python wants. > > I think someone fixed readline to work with Darwin's readline, at > least in theory. The machine is likely lacking the readline development headers. In any case, please don't focus on the skipped tests. What's important is the failed tests. Bill, Michael, it would be nice if you could investigate a bit more on these failures. Regards Antoine. From db3l.net at gmail.com Thu Apr 29 22:50:27 2010 From: db3l.net at gmail.com (David Bolen) Date: Thu, 29 Apr 2010 16:50:27 -0400 Subject: [Python-Dev] please take a look at buildbot result References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <136.1272559248@parc.com> Message-ID: Bill Janssen writes: >Ronald Oussoren wrote: >> As Antoine noted the test failures are unexpected, could you check if >> the tests pass if you do the build and testrun manually? > What about readline? Darwin doesn't have it (or rather, it has a > different one). Why is that 'skip' unexpected? For what it's worth, for the libraries, I used the build-installer script to build the same versions of libraries that it uses for the binary installer, and installed them in /usr/local for my buildbot to use, until there's enough time to have the buildbot build such local libraries within its own tree. That at least matches up the external dependencies for buildbot builds with that used by the eventual binary installer. For my buildbot, generally the only unexpected skips are for test_bsddb185 (which feels right to me - I don't have that version of the library, nor do I think has the binary installer) and test_ioctl (which I have no idea on yet). >> IMHO it would be better to do a framework build for at least some of >> the OSX buildbots because that's what is in the binary installer for >> OSX. > > Yes, that sounds good to me, too. But how do we make that a standard > test so that appropriately enabled buildbot slaves will do it > automatically? Something that gets configured in the build master? This came up in the earlier discussion, and while I do still think that's a better long term approach, I suspect that with respect to test coverage and code generation the non-framework build for the interpreter is a fair representation for testing. I suspect much of this is just a builder change on the master (to use the right Makefile targets for generating the framework build), but just given experience to date getting the binary installer building under buildbot, there may be some unexpected environmental stuff. Ideally we'd work up a way to do a universal framework build (though I'm not sure what if any extra support may be needed to have the system use the framework if not installed in a standard location), and then for full testing, maybe even use the makefile target that tests both the Intel and PPC path (the latter via Rosetta on an Intel system). What I'm not sure about at the moment is how much different in terms of testing such a setup might be (if, for example, any extra work is needed to be able to test a framework build while still local in the buildbot tree and not in a normal framework location), so how much energy is worth putting into it, especially if that might use resource among those able to resolve some of the open code issues. For my part, if there's free cycles I'd personally rather address the external library issue first, as opposed to the framework build stuff, since that feels more fragile to me (in terms of the buildbot environment for the Mac) at the moment. Over the past few weeks I'm also fairly sure that I'm finding stranded python test processes (not too dissimilar as on the Windows buildbots) that are bogging down the buildbot and thus more detrimental to ongoing buildbot operation than the lack of framework builds, so there may be other unknown issues more valuable to hit as we get more experience. -- David From ncoghlan at gmail.com Thu Apr 29 23:17:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Apr 2010 07:17:02 +1000 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: <20100429200533.18CE720087A@kimball.webabinitio.net> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> <20100429200533.18CE720087A@kimball.webabinitio.net> Message-ID: <4BD9F74E.4020907@gmail.com> R. David Murray wrote: > On Thu, 29 Apr 2010 14:11:57 -0400, Terry Reedy wrote: >> On 4/27/2010 5:14 PM, "Martin v. Loewis" wrote: >>> You only have resolutions on closed issues, >> Only for 'closed' or also for 'pending' and 'languishing'? >> If pending were implemented to mean 'auto close in x days', then >> Resolution would need to be set when Status is set to 'pending'. > > Definitely. Resolution should always be set for Pending, IMO. Note that I'll occasionally use "pending" to mean "committed to at least one branch, still need to port/block remaining branches". (e.g. I did it this week, since 2.7b2 was higher priority than the other branches which don't have near term binary releases coming up). I've seen others use it that way as well. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Thu Apr 29 23:20:56 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 29 Apr 2010 23:20:56 +0200 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> Message-ID: <4BD9F838.90906@v.loewis.de> >> You only have resolutions on closed issues, > > Only for 'closed' or also for 'pending' and 'languishing'? Probably for all of them. I don't understand languishing, though (neither as an English word, nor as a status). > 'Languishing' is new (its first use was two months ago). Is it a form of > 'open' or of 'closed'? I have no idea. > This needs to be explicitly stated in the doc if you want it followed. I personally don't - I don't think it matters that much to have a resolution field if you also post a message explaining why the issue was closed. I'm just reporting what it was meant for originally. > I never noticed this. there are also pop-up boxed for Stage, Priority, > and Keywords. There should also be a pop-up for Resolution. > > And the pop-up tables should also be in the doc. Contributions welcome. Regards, Martin From tseaver at palladion.com Fri Apr 30 00:07:24 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 29 Apr 2010 18:07:24 -0400 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: <4BD9F838.90906@v.loewis.de> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> <4BD9F838.90906@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> You only have resolutions on closed issues, >> Only for 'closed' or also for 'pending' and 'languishing'? > > Probably for all of them. I don't understand languishing, though > (neither as an English word, nor as a status). > >> 'Languishing' is new (its first use was two months ago). Is it a form of >> 'open' or of 'closed'? > > I have no idea. Open, definitely: it implies "suffering due to neglect", combining senses 2 and 5 from this page: http://en.wiktionary.org/wiki/languish Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvaAxUACgkQ+gerLs4ltQ6lIwCePWePnXRfHbvRcx3Ih6rV7AhQ qiYAn2V5yR/PjIb+WSr/ILhO5p3WGHoO =XWHS -----END PGP SIGNATURE----- From janssen at parc.com Fri Apr 30 02:27:44 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 29 Apr 2010 17:27:44 PDT Subject: [Python-Dev] =?utf-8?q?please_take_a_look_at_buildbot_result_=5Bw?= =?utf-8?q?as=3A_Broken=09link_to_download_=28Mac_OS_X=29=5D?= In-Reply-To: References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> <20100429184733.3009420087A@kimball.webabinitio.net> Message-ID: <32765.1272587264@parc.com> Antoine Pitrou wrote: > R. David Murray bitdance.com> writes: > > > > On Thu, 29 Apr 2010 11:28:47 -0700, Bill Janssen wrote: > > > 8 skips unexpected on darwin: > > > test_aepack test_applesingle test_gdb test_macos test_macostools > > > test_readline test_scriptpackages test_ttk_guionly > > > > > > Why is the skip of "test_readline" unexpected on darwin? The readline > > > on Darwin isn't what Python wants. > > > > I think someone fixed readline to work with Darwin's readline, at > > least in theory. > > The machine is likely lacking the readline development headers. > In any case, please don't focus on the skipped tests. What's important is the > failed tests. > Bill, Michael, it would be nice if you could investigate a bit more on these > failures. Well, test_grp test is failing because it assumes that group IDs are unique, which is not guaranteed by getgrent(). And apparently not true at PARC. test_posix fails on some group-related thing, too: the use of initgroups(), which is supposed to fail, succeeds. test_pwd fails because the fake uid chosen for one test is a real UID for a PARC user. So these are really problems with the test being based on an insufficiently general model of the real world. Bill From filipz at 3g.nec.com.au Fri Apr 30 03:11:59 2010 From: filipz at 3g.nec.com.au (Filip) Date: Fri, 30 Apr 2010 11:11:59 +1000 Subject: [Python-Dev] [melbourne-pug] array matching In-Reply-To: <83538.68421.qm@web114307.mail.gq1.yahoo.com> References: <83538.68421.qm@web114307.mail.gq1.yahoo.com> Message-ID: <98A0E3F45E944347B711EEB64C76FC94@spoon> You could use dictionary of lists test = { A: [1, 3], B: [2, 4] } print test[A] [1, 3] test[A].append(5) print test[A] [1, 3, 5] Cheers, Fil _____ From: melbourne-pug-bounces+filipz=3g.nec.com.au at python.org [mailto:melbourne-pug-bounces+filipz=3g.nec.com.au at python.org] On Behalf Of Bill Jordan Sent: Friday, 30 April 2010 3:58 AM To: python-dev at python.org; python-list at python.org; bangpypers at python.org; melbourne-pug at python.org; portland at python.org Subject: [melbourne-pug] array matching Hey guys, I am sorry if this is not the right list to post some questions. I have a simple question please and would appreciate some answers as I am new to Python. I have 2 D array: test = [[A,1],[B,2],[A,3][B,4]] I want to arrang this array in different arrays so each one will have what is attached to. For example I want the output: A =[1,3] and B=[2,4] Thanks, Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Fri Apr 30 04:48:18 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Apr 2010 22:48:18 -0400 Subject: [Python-Dev] The Meaning of Resolotion (Re: bug tracker permissions request) In-Reply-To: <4BD9F74E.4020907@gmail.com> References: <20100427151447.B5F492001AD@kimball.webabinitio.net> <20100427161634.3C8C81AB5FD@kimball.webabinitio.net> <4BD753CF.6070500@v.loewis.de> <20100429200533.18CE720087A@kimball.webabinitio.net> <4BD9F74E.4020907@gmail.com> Message-ID: <20100430024818.69630200AFC@kimball.webabinitio.net> On Fri, 30 Apr 2010 07:17:02 +1000, Nick Coghlan wrote: > R. David Murray wrote: > > On Thu, 29 Apr 2010 14:11:57 -0400, Terry Reedy wrote: > >> On 4/27/2010 5:14 PM, "Martin v. Loewis" wrote: > >>> You only have resolutions on closed issues, > >> Only for 'closed' or also for 'pending' and 'languishing'? > >> If pending were implemented to mean 'auto close in x days', then > >> Resolution would need to be set when Status is set to 'pending'. > > > > Definitely. Resolution should always be set for Pending, IMO. > > Note that I'll occasionally use "pending" to mean "committed to at least > one branch, still need to port/block remaining branches". (e.g. I did it > this week, since 2.7b2 was higher priority than the other branches which > don't have near term binary releases coming up). > > I've seen others use it that way as well. Yes, I have noticed that several people do this. This will stop working if we implement autoclose. Other people just leave the issue open and change the targeted versions instead. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Fri Apr 30 04:52:03 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Apr 2010 22:52:03 -0400 Subject: [Python-Dev] =?utf-8?q?please_take_a_look_at_buildbot_result_=5Bw?= =?utf-8?q?as=3A_Broken=09link_to_download_=28Mac_OS_X=29=5D?= In-Reply-To: <32765.1272587264@parc.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> <20100429184733.3009420087A@kimball.webabinitio.net> <32765.1272587264@parc.com> Message-ID: <20100430025203.8901820091B@kimball.webabinitio.net> On Thu, 29 Apr 2010 17:27:44 -0700, Bill Janssen wrote: > Well, test_grp test is failing because it assumes that group IDs are > unique, which is not guaranteed by getgrent(). And apparently not true > at PARC. > > test_posix fails on some group-related thing, too: the use of > initgroups(), which is supposed to fail, succeeds. > > test_pwd fails because the fake uid chosen for one test is a real UID > for a PARC user. > > So these are really problems with the test being based on an > insufficiently general model of the real world. Could you file bugs for these, please? The first and last others should be able to replicate easily; the middle one will probably require your help on the machine in question to debug. -- R. David Murray www.bitdance.com From ronaldoussoren at mac.com Fri Apr 30 07:28:53 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 30 Apr 2010 07:28:53 +0200 Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <20100429184733.3009420087A@kimball.webabinitio.net> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> <20100429184733.3009420087A@kimball.webabinitio.net> Message-ID: <8DE8AC44-8EBB-47FD-876A-5C421F38DAD1@mac.com> On 29 Apr, 2010, at 20:47, R. David Murray wrote: > On Thu, 29 Apr 2010 11:28:47 -0700, Bill Janssen wrote: >> 8 skips unexpected on darwin: >> test_aepack test_applesingle test_gdb test_macos test_macostools >> test_readline test_scriptpackages test_ttk_guionly >> >> Why is the skip of "test_readline" unexpected on darwin? The readline >> on Darwin isn't what Python wants. > > I think someone fixed readline to work with Darwin's readline, at > least in theory. The readline extension works with Darwin's readline emulation, but only when the deployment target is 10.5 or later. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From ronaldoussoren at mac.com Fri Apr 30 17:32:01 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 30 Apr 2010 17:32:01 +0200 Subject: [Python-Dev] please take a look at buildbot result [was: Broken link to download (Mac OS X)] In-Reply-To: <22670.1272565727@parc.com> References: <4BA62546.10906@python.org> <4BC54F4F.4090307@v.loewis.de> <4BC61278.7020108@v.loewis.de> <19399.11323.946604.992452@montanaro.dyndns.org> <96960.1272551962@parc.com> <28CD4E46-95F8-4191-AC58-FD9CE3578F07@mac.com> <4BD9C354.2010902@voidspace.org.uk> <22670.1272565727@parc.com> Message-ID: On 29 Apr, 2010, at 20:28, Bill Janssen wrote: > Michael Foord wrote: > >> Well - I have nine failing tests on trunk for Mac OS X with Snow Leopard. >> >> 9 tests failed: >> test_cmd_line test_imp test_import test_posix test_pydoc >> test_runpy test_urllib2 test_urllib2_localnet test_warnings >> >> I believe that Victor has ensured that the buildbot failures have open >> issues against, I'm just going to check that the failures I see are >> the same. > > Just built on Intel/Leopard with the trunk (almost vanilla, except for > _ssl.c) and tested. Here's what I configured: > > ./configure --prefix=/local/python/trunk --disable-universalsdk --disable-framework --disable-toolbox-glue --with-pydebug > > ... > 8 skips unexpected on darwin: > test_aepack test_applesingle test_gdb test_macos test_macostools > test_readline test_scriptpackages test_ttk_guionly > > Why is the skip of "test_readline" unexpected on darwin? The readline > on Darwin isn't what Python wants. The skip of test_readline is unexpected because readline should have been build with the options above and tests should have worked. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From status at bugs.python.org Fri Apr 30 18:08:52 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 30 Apr 2010 18:08:52 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100430160852.DEE9E78130@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-04-23 - 2010-04-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2638 open (+41) / 17759 closed (+27) / 20397 total (+68) Open issues with patches: 1081 Average duration of open issues: 724 days. Median duration of open issues: 494 days. Open Issues Breakdown open 2620 (+41) languishing 10 ( +0) pending 7 ( +0) Issues Created Or Reopened (71) _______________________________ "Report bug" links 2010-04-24 CLOSED http://bugs.python.org/issue2823 reopened techtonik Silence DeprecationWarning by default 2010-04-25 http://bugs.python.org/issue7319 reopened benjamin.peterson patch Small mistake in tutorial web page 2010-04-23 CLOSED http://bugs.python.org/issue8511 created mdcowles os.execv*e(): fix formatting of the environment variables (Pyt 2010-04-23 CLOSED http://bugs.python.org/issue8512 created haypo patch subprocess: support bytes program name 2010-04-23 http://bugs.python.org/issue8513 created haypo patch Create fsencode() and fsdecode() functions in os.path 2010-04-23 http://bugs.python.org/issue8514 created haypo patch idle "Run Module" (F5) does not set __file__ variable 2010-04-24 http://bugs.python.org/issue8515 created dangyogi patch Speed difference between Python 2.5 and 2.6 during filling bsd 2010-04-24 http://bugs.python.org/issue8516 created PeterL Apple Style Guide link is broken in the "Documenting Python" c 2010-04-24 CLOSED http://bugs.python.org/issue8517 created brandon-rhodes small typo in http://docs.python.org/howto/doanddont.html 2010-04-24 CLOSED http://bugs.python.org/issue8518 created Adri??n.Deccico [patch] doc: termios and ioctl reference links 2010-04-24 http://bugs.python.org/issue8519 created techtonik patch test doc issue 2010-04-24 CLOSED http://bugs.python.org/issue8520 created georg.brandl Allow some winreg functions to accept keyword arguments 2010-04-24 http://bugs.python.org/issue8521 created brian.curtin patch enhacement proposal in howto/doanddont 2010-04-24 CLOSED http://bugs.python.org/issue8522 created Adri??n.Deccico patch shutil.rmtree and os.listdir cannot recover on error condition 2010-04-24 http://bugs.python.org/issue8523 created rubenlm easy SSL sockets do not retain the parent socket's attributes 2010-04-24 http://bugs.python.org/issue8524 created pitrou Small enhancement to help() 2010-04-24 http://bugs.python.org/issue8525 created robcliffe msilib doesn't support multiple CAB instances in same installe 2010-04-25 http://bugs.python.org/issue8526 created janssen easy [PEP 3147] compileall.compile_dir() called multiple times crea 2010-04-25 CLOSED http://bugs.python.org/issue8527 created Arfrever patch typo in argparse documentation 2010-04-25 CLOSED http://bugs.python.org/issue8528 created akira subclassing builtin class (str, unicode, list...) needs to ove 2010-04-25 CLOSED http://bugs.python.org/issue8529 created flox Stringlib fastsearch can read beyond the front of an array 2010-04-25 http://bugs.python.org/issue8530 created alex patch test_ascii_formatd fails if ctypes module is missing 2010-04-25 CLOSED http://bugs.python.org/issue8531 created haypo Refinements to Python 3 New GIL 2010-04-26 CLOSED http://bugs.python.org/issue8532 created dabeaz patch regrtest: use backslashreplace error handler for stdout 2010-04-26 http://bugs.python.org/issue8533 created haypo patch multiprocessing not working from egg 2010-04-26 http://bugs.python.org/issue8534 created simonsteiner patch passing optimization flags to the linker required for builds w 2010-04-26 http://bugs.python.org/issue8535 created doko patch, needs review Support new features of ZLIB 1.2.4 2010-04-26 http://bugs.python.org/issue8536 created jcea easy Add ConfigureAction to argparse 2010-04-26 CLOSED http://bugs.python.org/issue8537 created eric.smith Add ConfigureAction to argparse 2010-04-26 http://bugs.python.org/issue8538 created eric.smith Add ConfigureAction to argparse 2010-04-26 CLOSED http://bugs.python.org/issue8539 created eric.smith Make Context._clamp public in decimal module 2010-04-26 http://bugs.python.org/issue8540 created mark.dickinson Test issue 2010-04-26 CLOSED http://bugs.python.org/issue8541 created loewis Another test issue 2010-04-26 CLOSED http://bugs.python.org/issue8542 created loewis asynchat documentation issues 2010-04-27 CLOSED http://bugs.python.org/issue8543 created giampaolo.rodola i didn't get rqrd output for programme in python?(plz... help) 2010-04-27 CLOSED http://bugs.python.org/issue8544 created krishnalolla i didn't get rqrd output for programme in python?(plz... help) 2010-04-27 CLOSED http://bugs.python.org/issue8545 created krishnalolla The parameter buffering in _pyio.open doesn't work the same as 2010-04-27 CLOSED http://bugs.python.org/issue8546 created pyfex patch unittest test discovery can fail when package under test is al 2010-04-27 http://bugs.python.org/issue8547 created michael.foord Building on CygWin 1.7: PATH_MAX redefined 2010-04-27 http://bugs.python.org/issue8548 created jbinder Modules/_ssl.c: extra comma breaks build on AIX 2010-04-27 CLOSED http://bugs.python.org/issue8549 created srid patch Expose SSL contexts 2010-04-27 http://bugs.python.org/issue8550 created pitrou Start argument for str.rfind used incorrectly 2010-04-27 CLOSED http://bugs.python.org/issue8551 created dtorp msilib can't create large CAB files 2010-04-27 http://bugs.python.org/issue8552 created janssen 2to3 breaks relative imports 2010-04-28 CLOSED http://bugs.python.org/issue8553 created jyasskin patch, patch suspicious comment in msilib.py/__init__.py 2010-04-28 http://bugs.python.org/issue8554 created janssen tkinter doesn't see _tkinter 2010-04-28 CLOSED http://bugs.python.org/issue8555 created py.user Confusing string formatting examples 2010-04-28 http://bugs.python.org/issue8556 created mcjeff patch subprocess PATH semantics and portability 2010-04-28 http://bugs.python.org/issue8557 created dabrahams StringIO().truncate causes zero-bytes in getvalue() 2010-04-28 CLOSED http://bugs.python.org/issue8558 created hpk test_gdb: test_strings() fails with ASCII locale 2010-04-28 http://bugs.python.org/issue8559 created haypo patch regrtest: add a minimal "progress bar" 2010-04-28 http://bugs.python.org/issue8560 created haypo patch Install .exes generated with distutils to not do a CRC check 2010-04-28 http://bugs.python.org/issue8561 created nateman1352 hasattr(open,'newlines') example gives incorrect results from 2010-04-28 http://bugs.python.org/issue8562 created adamnelson [PEP 3147] compileall.compile_file() creates empty __pycache__ 2010-04-28 CLOSED http://bugs.python.org/issue8563 created Arfrever patch Update documentation on doctest/unittest2 integration 2010-04-28 http://bugs.python.org/issue8564 created pfein patch Always run regrtest.py with -bb 2010-04-28 http://bugs.python.org/issue8565 created haypo patch 2to3 should run under python 2.5 2010-04-28 http://bugs.python.org/issue8566 created jyasskin patch, patch, needs review decimal module doesn't respect precedence rules for exceptiona 2010-04-29 http://bugs.python.org/issue8567 created mark.dickinson Test issue 2010-04-29 CLOSED http://bugs.python.org/issue8568 created mark.dickinson Upgrade OpenSSL in Windows builds 2010-04-29 http://bugs.python.org/issue8569 created pitrou 2to3 crashes with "AttributeError: 'int' object has no attribu 2010-04-29 CLOSED http://bugs.python.org/issue8570 created rom16384 zlib causes a SystemError when decompressing a chunk >1GB 2010-04-29 http://bugs.python.org/issue8571 created ddmitriev httplib getheader() throws error instead of default 2010-04-29 http://bugs.python.org/issue8572 created Walter.Woods easy Buggy _strerror in asyncore 2010-04-29 http://bugs.python.org/issue8573 created q94IjzUfnNoyv4c75mMw easy transient_internet() (test_support): use socket.setdefaulttime 2010-04-29 http://bugs.python.org/issue8574 created haypo patch Update/reorganize _winreg documentation 2010-04-30 http://bugs.python.org/issue8575 created brian.curtin patch, patch, needs review test_support.find_unused_port can cause socket conflicts on Wi 2010-04-30 http://bugs.python.org/issue8576 created pmoore buildbot test_distutils fails if srcdir != builddir 2010-04-30 http://bugs.python.org/issue8577 created ronaldoussoren patch PyWeakref_GetObject 2010-04-30 http://bugs.python.org/issue8578 created arigo URI parsing library 2010-04-29 http://bugs.python.org/issue1462525 reopened orsenthil patch Issues Now Closed (112) _______________________ uudecoding (uu.py) does not supprt base64, patch attached 977 days http://bugs.python.org/issue1027 tjreedy patch Carbon.CF memory management problem 960 days http://bugs.python.org/issue1155 tjreedy ssl module doesn't support non-blocking handshakes 931 days http://bugs.python.org/issue1251 pitrou patch warnings module bug: BytesWarning: str() on a bytes instance 901 days http://bugs.python.org/issue1404 loewis patch IDLE not working correctly on Windows (Py30a2/IDLE30a1) 867 days http://bugs.python.org/issue1601 loewis Execfile unable to take arguments beyond 255! 864 days http://bugs.python.org/issue1636 loewis Uses of SocketServer.BaseServer.shutdown have a race 770 days http://bugs.python.org/issue2302 pitrou patch, patch Backport parameter annotations 771 days http://bugs.python.org/issue2331 benjamin.peterson 26backport Backport PEP 3132 (extended iterable unpacking) 771 days http://bugs.python.org/issue2340 benjamin.peterson 26backport SO_REUSEADDR doesn't have the same semantics on Windows as on 753 days http://bugs.python.org/issue2550 pitrou patch, 26backport cmd.py always uses raw_input, even when another stdin is speci 752 days http://bugs.python.org/issue2571 tjreedy easy Remove carbon-specific code from binhex 731 days http://bugs.python.org/issue2715 mark.dickinson "Report bug" links 0 days http://bugs.python.org/issue2823 georg.brandl ssl module - should test for a wrong cert 670 days http://bugs.python.org/issue3212 pitrou patch When PyObject_CallMethod fails, refcount is incorrect 659 days http://bugs.python.org/issue3328 tjreedy "Suggest a change" link 96 days http://bugs.python.org/issue3341 georg.brandl Mac, 3.0 framework install error with fink cp 645 days http://bugs.python.org/issue3433 tjreedy case conversion problems in Turkish 627 days http://bugs.python.org/issue3536 tjreedy Provide a way to disable SSLv2 (or better yet, disable by defa 617 days http://bugs.python.org/issue3596 pitrou MacOS X framework install to non-standard directory fails 616 days http://bugs.python.org/issue3646 ronaldoussoren patch os.mknod missing on Solaris 583 days http://bugs.python.org/issue3928 jcea patch, needs review configure --with-threads on cygwin => crash on thread related 581 days http://bugs.python.org/issue3947 loewis patch [contextlib] Reverse order of locking 582 days http://bugs.python.org/issue3957 tjreedy C99 comments in Python 2.6 break build on AIX 6.1 573 days http://bugs.python.org/issue4025 r.david.murray Do not embed manifest files in *.pyd when compiling with MSVC 559 days http://bugs.python.org/issue4120 loewis patch SSL handshake fails after TCP connection in getpeername() 551 days http://bugs.python.org/issue4171 pitrou type() doesn't accept bases and dict keyword arguments 553 days http://bugs.python.org/issue4186 r.david.murray import and compile() do not treat trailing whitespace the same 540 days http://bugs.python.org/issue4262 tjreedy closing stdout in a child process on cygwin means that process 533 days http://bugs.python.org/issue4295 stutzbach ssl.SSLSocket timeout not working correctly when remote end is 450 days http://bugs.python.org/issue5103 pitrou patch ssl makefile never closes socket 435 days http://bugs.python.org/issue5238 pitrou patch Fix strftime on windows. 438 days http://bugs.python.org/issue5249 loewis patch 3.0.1 crashes in unicode path 436 days http://bugs.python.org/issue5273 loewis patch array.fromfile() fails to insert values when EOFError is raise 431 days http://bugs.python.org/issue5334 loewis patch Broken Py3.1 release build in Visual Studio 2005 423 days http://bugs.python.org/issue5407 tjreedy Failure in test_httpservers on Linux 410 days http://bugs.python.org/issue5494 tjreedy Optional extensions in setup.py 393 days http://bugs.python.org/issue5583 tarek patch Create alternative CObject API that is safe and clean 389 days http://bugs.python.org/issue5630 jcea patch _winreg.OpenKey() is documented with keyword arguments, but do 375 days http://bugs.python.org/issue5774 merwok patch, needs review test_unicode fails in wide unicode build 332 days http://bugs.python.org/issue6150 loewis filecmp.cmp can not compare two files from different OS with t 315 days http://bugs.python.org/issue6306 tjreedy httplib fails with HEAD requests to pages with "transfer-encod 313 days http://bugs.python.org/issue6312 orsenthil patch Improve bool TypeError message 292 days http://bugs.python.org/issue6453 loewis easy codecs.open on Win32 does not force binary mode 246 days http://bugs.python.org/issue6788 tjreedy OptionParser.allow_interspersed_args is undocumented 230 days http://bugs.python.org/issue6883 r.david.murray Socket error when launching IDLE 223 days http://bugs.python.org/issue6941 tjreedy test_io.py: codecs.IncrementalDecoder is sometimes None 209 days http://bugs.python.org/issue7027 haypo patch Missing uninstallation instructions for mac 201 days http://bugs.python.org/issue7107 ronaldoussoren Detect improper leading whitespace in C files for Vim 168 days http://bugs.python.org/issue7288 brett.cannon easy A number tests "crash" if python is compiled --without-threads 143 days http://bugs.python.org/issue7449 haypo patch, easy, needs review pipes.quote does not correctly escape ! 130 days http://bugs.python.org/issue7507 georg.brandl patch urllib2 request does not update content length after new add_d 54 days http://bugs.python.org/issue7540 loewis patch socket.connect() no longer works with AF_BLUETOOTH L2CAP socke 86 days http://bugs.python.org/issue7834 pitrou patch, easy Undocumented subprocess functions on Windows 81 days http://bugs.python.org/issue7838 brian.curtin patch, easy, needs review [PATCH] Drop "Computer" from "Apple Computer" in plistlib 9 days http://bugs.python.org/issue7852 ronaldoussoren patch urlparse.urlsplit mishandles novel schemes 75 days http://bugs.python.org/issue7904 tseaver Memory leak due to circular references in ssl.SSLSocket 66 days http://bugs.python.org/issue7943 pitrou patch Backport capsule object 61 days http://bugs.python.org/issue7992 jcea patch, buildbot ssl.get_server_certificate new line missing 51 days http://bugs.python.org/issue8086 pitrou patch, easy test_ftplib fails with OpenSSL 0.9.8m 44 days http://bugs.python.org/issue8108 pitrou patch, buildbot String interpolation with unicode subclass fails to call __str 45 days http://bugs.python.org/issue8128 eric.smith Incoherent error with keyword argument follow by unpacking ar 36 days http://bugs.python.org/issue8177 benjamin.peterson add unpack_archive to shutil 26 days http://bugs.python.org/issue8295 tarek HTMLparser does not handle call to handle_data when a tag cont 19 days http://bugs.python.org/issue8319 orsenthil easy improve regrtest command line help 20 days http://bugs.python.org/issue8325 r.david.murray patch, easy, needs review os.execvpe() doesn't support surrogates in env 12 days http://bugs.python.org/issue8391 haypo patch dict constructor allows invalid identifiers in **kwargs 8 days http://bugs.python.org/issue8419 benjamin.peterson patch syslog.syslog('msg') logs message with ident "python". 5 days http://bugs.python.org/issue8451 r.david.murray patch, easy, needs review tarfile creates tarballs with execute permissions set 10 days http://bugs.python.org/issue8464 lars.gustaebel patch subprocess: surrogates of the error message (Python implementa 3 days http://bugs.python.org/issue8467 haypo patch build-installer fix for setIcon when no script path specified 10 days http://bugs.python.org/issue8476 ronaldoussoren patch socket's send can raise errno 35 under OS X, which causes prob 6 days http://bugs.python.org/issue8493 mdcowles test_gdb: use utf8+surrogateescape charset? 2 days http://bugs.python.org/issue8495 haypo patch bsddb databases in python 2.6 are not compatible with python 2 0 days http://bugs.python.org/issue8504 loewis 2to3 fix_future.py removes __future__ imports, but should it? 1 days http://bugs.python.org/issue8505 barry Small mistake in tutorial web page 0 days http://bugs.python.org/issue8511 ezio.melotti os.execv*e(): fix formatting of the environment variables (Pyt 2 days http://bugs.python.org/issue8512 haypo patch Apple Style Guide link is broken in the "Documenting Python" c 0 days http://bugs.python.org/issue8517 georg.brandl small typo in http://docs.python.org/howto/doanddont.html 0 days http://bugs.python.org/issue8518 georg.brandl test doc issue 0 days http://bugs.python.org/issue8520 georg.brandl enhacement proposal in howto/doanddont 1 days http://bugs.python.org/issue8522 georg.brandl patch [PEP 3147] compileall.compile_dir() called multiple times crea 2 days http://bugs.python.org/issue8527 barry patch typo in argparse documentation 0 days http://bugs.python.org/issue8528 georg.brandl subclassing builtin class (str, unicode, list...) needs to ove 0 days http://bugs.python.org/issue8529 flox test_ascii_formatd fails if ctypes module is missing 0 days http://bugs.python.org/issue8531 haypo Refinements to Python 3 New GIL 0 days http://bugs.python.org/issue8532 pitrou patch Add ConfigureAction to argparse 0 days http://bugs.python.org/issue8537 eric.smith Add ConfigureAction to argparse 0 days http://bugs.python.org/issue8539 eric.smith Test issue 0 days http://bugs.python.org/issue8541 loewis Another test issue 0 days http://bugs.python.org/issue8542 loewis asynchat documentation issues 3 days http://bugs.python.org/issue8543 giampaolo.rodola i didn't get rqrd output for programme in python?(plz... help) 0 days http://bugs.python.org/issue8544 orsenthil i didn't get rqrd output for programme in python?(plz... help) 0 days http://bugs.python.org/issue8545 georg.brandl The parameter buffering in _pyio.open doesn't work the same as 1 days http://bugs.python.org/issue8546 benjamin.peterson patch Modules/_ssl.c: extra comma breaks build on AIX 0 days http://bugs.python.org/issue8549 pitrou patch Start argument for str.rfind used incorrectly 0 days http://bugs.python.org/issue8551 r.david.murray 2to3 breaks relative imports 0 days http://bugs.python.org/issue8553 jyasskin patch, patch tkinter doesn't see _tkinter 0 days http://bugs.python.org/issue8555 py.user StringIO().truncate causes zero-bytes in getvalue() 0 days http://bugs.python.org/issue8558 pitrou [PEP 3147] compileall.compile_file() creates empty __pycache__ 1 days http://bugs.python.org/issue8563 Arfrever patch Test issue 0 days http://bugs.python.org/issue8568 mark.dickinson 2to3 crashes with "AttributeError: 'int' object has no attribu 0 days http://bugs.python.org/issue8570 benjamin.peterson cPickle does relative import 2681 days http://bugs.python.org/issue658693 tjreedy ftplib.retrbinary fails when called from retrlines callback 2510 days http://bugs.python.org/issue751758 giampaolo.rodola PyMarshal_ReadLastObjectFromFile 2483 days http://bugs.python.org/issue770280 mark.dickinson patch MacPython installer fails on UFS filesystem 2457 days http://bugs.python.org/issue785031 tjreedy Carbon.File fails if server vol is mounted after launch 2092 days http://bugs.python.org/issue1004810 tjreedy bdist_deb - Debian packager 436 days http://bugs.python.org/issue1054967 barry patch configure: refuses setgroups 1848 days http://bugs.python.org/issue1178510 tjreedy use PyOS_ReadlineFunctionPointer in non-interractive input 1614 days http://bugs.python.org/issue1367628 tjreedy patch test test_capi crashed -- thread.error: can't allocate lock 1470 days http://bugs.python.org/issue1473979 tjreedy Python 2.5.1 fails to build on AIX 1016 days http://bugs.python.org/issue1756343 tjreedy Top Issues Most Discussed (10) ______________________________ 28 Convoy effect with I/O bound threads and New GIL 73 days open http://bugs.python.org/issue7946 17 bdist_deb - Debian packager 436 days closed http://bugs.python.org/issue1054967 15 socket's send can raise errno 35 under OS X, which causes probl 6 days closed http://bugs.python.org/issue8493 14 Create fsencode() and fsdecode() functions in os.path 7 days open http://bugs.python.org/issue8514 11 test_distutils fails if srcdir != builddir 0 days open http://bugs.python.org/issue8577 11 httplib getheader() throws error instead of default 1 days open http://bugs.python.org/issue8572 9 [PEP 3147] compileall.compile_file() creates empty __pycache__ 1 days closed http://bugs.python.org/issue8563 9 shutil.rmtree and os.listdir cannot recover on error conditions 6 days open http://bugs.python.org/issue8523 9 update to autoconf2.65 7 days open http://bugs.python.org/issue8510 9 datetime's comparison methods do not return NotImplemented when 66 days open http://bugs.python.org/issue8005 From barry at python.org Fri Apr 30 21:09:18 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 30 Apr 2010 15:09:18 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> Message-ID: <20100430150918.308984aa@heresy> On Apr 28, 2010, at 09:22 AM, Dirkjan Ochtman wrote: >On Tue, Apr 27, 2010 at 23:55, Nick Coghlan wrote: >> I believe the more important part of Barry's suggested change here is >> requiring a link to the archived message (usually from python-dev) where >> the PEP was accepted (be it directly by you as BDFL, or by consensus >> from a "sufficient" number of core developers). This will likely also >> help with reminding people to announce on python-dev when PEPs are >> accepted by consensus (or by you) somewhere like PyCon or a sprint. > >Though maybe it should be called Conclusion instead of Accepted and >used for Rejected PEPs, as well? Good point. What do you think about 'Resolution'? As to Guido's point about the decision making process, Nick's right. I just want to make sure we can capture the resolution in the PEP, be it by BDFL pronouncement or "hey, silence is acceptance" email. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Fri Apr 30 21:51:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Apr 2010 21:51:42 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100430150918.308984aa@heresy> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> Message-ID: <4BDB34CE.9080804@v.loewis.de> > As to Guido's point about the decision making process, Nick's right. I just > want to make sure we can capture the resolution in the PEP, be it by BDFL > pronouncement or "hey, silence is acceptance" email. I don't think "silence is acceptance" will work out in practice. For issues where a PEP was written in the first place, somebody will *always* object, and forever so, hoping that what he considers a mistake will not be done. The advantage of Guido acting as BDFL was that somebody would make an decision ultimately, one which both proponents and opponents of the PEP would accept. Without a BDFL, I think we need a committee to make decisions, e.g. by majority vote amongst committers. Regards, Martin From steve at holdenweb.com Fri Apr 30 22:28:28 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 30 Apr 2010 16:28:28 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB34CE.9080804@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> Message-ID: <4BDB3D6C.4050909@holdenweb.com> Martin v. L?wis wrote: >> As to Guido's point about the decision making process, Nick's right. I just >> want to make sure we can capture the resolution in the PEP, be it by BDFL >> pronouncement or "hey, silence is acceptance" email. > > I don't think "silence is acceptance" will work out in practice. For > issues where a PEP was written in the first place, somebody will > *always* object, and forever so, hoping that what he considers a mistake > will not be done. The advantage of Guido acting as BDFL was that > somebody would make an decision ultimately, one which both proponents > and opponents of the PEP would accept. > > Without a BDFL, I think we need a committee to make decisions, e.g. by > majority vote amongst committers. Couldn't we just go with the FLUFL? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From steve at holdenweb.com Fri Apr 30 22:28:28 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 30 Apr 2010 16:28:28 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB34CE.9080804@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> Message-ID: <4BDB3D6C.4050909@holdenweb.com> Martin v. L?wis wrote: >> As to Guido's point about the decision making process, Nick's right. I just >> want to make sure we can capture the resolution in the PEP, be it by BDFL >> pronouncement or "hey, silence is acceptance" email. > > I don't think "silence is acceptance" will work out in practice. For > issues where a PEP was written in the first place, somebody will > *always* object, and forever so, hoping that what he considers a mistake > will not be done. The advantage of Guido acting as BDFL was that > somebody would make an decision ultimately, one which both proponents > and opponents of the PEP would accept. > > Without a BDFL, I think we need a committee to make decisions, e.g. by > majority vote amongst committers. Couldn't we just go with the FLUFL? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From guido at python.org Fri Apr 30 23:25:02 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Apr 2010 14:25:02 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB3D6C.4050909@holdenweb.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: On Fri, Apr 30, 2010 at 1:28 PM, Steve Holden wrote: > Martin v. L?wis wrote: >>> As to Guido's point about the decision making process, Nick's right. ?I just >>> want to make sure we can capture the resolution in the PEP, be it by BDFL >>> pronouncement or "hey, silence is acceptance" email. >> >> I don't think "silence is acceptance" will work out in practice. For >> issues where a PEP was written in the first place, somebody will >> *always* object, and forever so, hoping that what he considers a mistake >> will not be done. The advantage of Guido acting as BDFL was that >> somebody would make an decision ultimately, one which both proponents >> and opponents of the PEP would accept. >> >> Without a BDFL, I think we need a committee to make decisions, e.g. by >> majority vote amongst committers. > > Couldn't we just go with the FLUFL? IIRC in the IETF this is done by the committee chair. I think it's a good idea to have this be a single person to avoid endless indecision. > regards > ?Steve > -- > Steve Holden ? ? ? ? ? +1 571 484 6266 ? +1 800 494 3119 > See PyCon Talks from Atlanta 2010 ?http://pycon.blip.tv/ > Holden Web LLC ? ? ? ? ? ? ? ? http://www.holdenweb.com/ > UPCOMING EVENTS: ? ? ? ?http://holdenweb.eventbrite.com/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Fri Apr 30 23:35:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Apr 2010 23:35:34 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB3D6C.4050909@holdenweb.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: <4BDB4D26.3040904@v.loewis.de> >> Without a BDFL, I think we need a committee to make decisions, e.g. by >> majority vote amongst committers. > > Couldn't we just go with the FLUFL? Not sure whether that's a serious proposal (April 1 is already some days back now). As a starting point, Barry would have to indicate whether he is interested in that role. Regards, Martin From steve at holdenweb.com Fri Apr 30 23:51:04 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 30 Apr 2010 17:51:04 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB4D26.3040904@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDB4D26.3040904@v.loewis.de> Message-ID: <4BDB50C8.1090205@holdenweb.com> Martin v. L?wis wrote: >>> Without a BDFL, I think we need a committee to make decisions, e.g. by >>> majority vote amongst committers. >> Couldn't we just go with the FLUFL? > > Not sure whether that's a serious proposal (April 1 is already some days > back now). As a starting point, Barry would have to indicate whether he > is interested in that role. > If he isn't then we can depose him and replace him with a puppet. regards Steve PS: Barry: sorry I can't make the gig tonight. Hope it went well ... -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/